<div dir="ltr">should i change all server <span style="color:rgb(80,0,80)">nsslapd-db-deadlock-policy: 6 in config  for problematic server only or all servers need restart?</span></div><div class="gmail_extra"><br><br><div class="gmail_quote">
2014-07-11 15:53 GMT+08:00  <span dir="ltr"><<a href="mailto:barrykfl@gmail.com" target="_blank">barrykfl@gmail.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div>At server 2 there is a error:</div><div><br></div><div><br></div>[10/Jul/2014:12:29:59 +0800] NSMMReplicationPlugin - agmt="cn=<a href="http://meToserver1.abc.com" target="_blank">meToserver1.abc.com</a>" (central:389): Replication bind with GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Credentials cache file '/tmp/krb5cc_494' not found))<br>

</div><div class="gmail_extra"><br><br><div class="gmail_quote">2014-07-11 10:26 GMT+08:00  <span dir="ltr"><<a href="mailto:barrykfl@gmail.com" target="_blank">barrykfl@gmail.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">

<div dir="ltr"><div>Yes , </div><div>still get "cant contact ldap server" after upgrading both    servers.</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">2014-07-10 23:18 GMT+08:00 Rich Megginson <span dir="ltr"><<a href="mailto:rmeggins@redhat.com" target="_blank">rmeggins@redhat.com</a>></span>:<br>


<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
  
    
  
  <div bgcolor="#FFFFFF" text="#000000">
    <div>On 07/10/2014 09:15 AM,
      <a href="mailto:barrykfl@gmail.com" target="_blank">barrykfl@gmail.com</a> wrote:<br>
    </div>
    <blockquote type="cite">
      <p dir="ltr">But any hint that server 2 say cant contact ldap
        server if type ipa command?</p>
    </blockquote>
    <br>
    Please keep replies on list.<br>
    <br>
    You still get "cant contact ldap server" after upgrading both
    servers?<br>
    <br>
    <blockquote type="cite">
      <div class="gmail_quote">2014/7/10 下午10:25 於 "Rich Megginson" <<a href="mailto:rmeggins@redhat.com" target="_blank">rmeggins@redhat.com</a>>
        寫道:<br type="attribution">
        <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid">
          <div bgcolor="#FFFFFF" text="#000000">
            <div>On 07/10/2014 01:14 AM, <a href="mailto:barrykfl@gmail.com" target="_blank">barrykfl@gmail.com</a>
              wrote:<br>
            </div>
            <blockquote type="cite">
              <div dir="ltr">Tried and now two version same ....but seem
                same situation.
                <div><br>
                </div>
                <div>i found a related error log that server1 has
                  account after added user but not replicated to
                  server2. Is it too fast on UI clicking ? as i exp once
                  that click very</div>
                <div>fast twice add and edit user may cause server 2 no
                  record.</div>
                <div><br>
                </div>
                <div><br>
                </div>
                <div>
                  <div>[10/Jul/2014:14:20:01 +0800]
                    NSMMReplicationPlugin - changelog program -
                    _cl5WriteOperationTxn: retry (49) the transaction
                    (csn=53be3097000000040000) failed (rc=-30994
                    (DB_LOCK_DEADLOCK: Locker killed to resolve a
                    deadlock))</div>
                  <div>[10/Jul/2014:14:20:01 +0800]
                    NSMMReplicationPlugin - changelog program -
                    _cl5WriteOperationTxn: failed to write entry with
                    csn (53be3097000000040000); db error - -30994
                    DB_LOCK_DEADLOCK: Locker killed to resolve a
                    deadlock</div>
                  <div>[10/Jul/2014:14:20:01 +0800]
                    NSMMReplicationPlugin - write_changelog_and_ruv:
                    can't add a change for
                    uid=xuehuimei,cn=users,cn=accounts,dc=abc,dc=com
                    (uniqid: 1300de84-07fa11e4-b3ddf885-593f3a7a,
                    optype: 16) to changelog csn 53be3097000000040000</div>
                  <div>[10/Jul/2014:14:56:51 +0800]
                    NSMMReplicationPlugin - changelog program -
                    _cl5WriteOperationTxn: retry (49) the transaction
                    (csn=53be3939000000040000) failed (rc=-30994
                    (DB_LOCK_DEADLOCK: Locker killed to resolve a
                    deadlock))</div>
                  <div>[10/Jul/2014:14:56:51 +0800]
                    NSMMReplicationPlugin - changelog program -
                    _cl5WriteOperationTxn: failed to write entry with
                    csn (53be3939000000040000); db error - -30994
                    DB_LOCK_DEADLOCK: Locker killed to resolve a
                    deadlock</div>
                  <div>[10/Jul/2014:14:56:51 +0800]
                    NSMMReplicationPlugin - write_changelog_and_ruv:
                    can't add a change for
                    uid=websubcon04,cn=users,cn=accounts,dc=abc,dc=com
                    (uniqid: 3e39fc81-07ff11e4-b3ddf885-593f3a7a,
                    optype: 16) to changelog csn 53be3939000000040000</div>
                </div>
              </div>
            </blockquote>
            <br>
            This looks like <a href="https://fedorahosted.org/389/ticket/47409" target="_blank">https://fedorahosted.org/389/ticket/47409</a>
            and <a href="https://bugzilla.redhat.com/show_bug.cgi?id=979169" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=979169</a><br>
            <br>
            Cause: Under certain conditions, with a mix of concurrent
            search and update and outgoing replication operations, there
            will be deadlocks in the changelog db, leading to error
            messages like this:<br>
            NSMMReplicationPlugin - changelog program -
            _cl5WriteOperationTxn: failed to write entry with csn
            (XXXXXXX); db error - -30994 DB_LOCK_DEADLOCK: Locker killed
            to resolve a deadlock<br>
            This is caused by a deadlock between the changelog readers,
            writers, and main database writers.<br>
            <br>
            Consequence: Update operations will fail with the above
            error message in the directory server errors log.<br>
            <br>
            Fix: A new configuration parameter is introduced:<br>
            dn: cn=config,cn=ldbm database,cn=plugins,cn=config<br>
            nsslapd-db-deadlock-policy: 9<br>
            <br>
            With the default policy 9 (DB_LOCK_YOUNGEST), the last
            locker gets killed when there is a deadlock.  In the case
            that this is the changelog writer, the write will fail, and
            the entire update will fail.<br>
            <br>
            Users who frequently see the above errors in the errors log
            are advised to change this setting to 6 (DB_LOCK_MINWRITE)
            will which instead kill the locker that has the fewest write
            locks (that is, the changelog reader).  The changelog reader
            code has been changed to handle this deadlock condition and
            retry.  The setting can be changed like this:<br>
            <br>
            ldapmodify -x -D "cn=directory manager" -W <<EOF<br>
            dn: cn=config,cn=ldbm database,cn=plugins,cn=config<br>
            changetype: modify<br>
            replace: nsslapd-db-deadlock-policy<br>
            nsslapd-db-deadlock-policy: 6<br>
            EOF<br>
            <br>
            You may ask why the default is not changed to 6.  The answer
            is that the setting will apply to _all_ threads, so that
            changing this setting could cause regular search requests to
            fail, if the directory server is under a heavy update load. 
            In our testing, we did not see this happen, but we cannot
            guarantee that changing this value to 6 will not impact
            regular search requests.<br>
            <br>
            Result: After changing nsslapd-db-deadlock-policy to 6,
            updates will succeed and no longer cause errors like the
            above.<br>
            <br>
            <br>
            <blockquote type="cite">
              <div dir="ltr">
                <div> </div>
              </div>
              <div class="gmail_extra"><br>
                <br>
                <div class="gmail_quote">2014-07-10 10:40 GMT+08:00 Rich
                  Megginson <span dir="ltr"><<a href="mailto:rmeggins@redhat.com" target="_blank">rmeggins@redhat.com</a>></span>:<br>
                  <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid">
                    <div bgcolor="#FFFFFF" text="#000000">
                      <div>
                        <div>On 07/09/2014 08:36 PM, <a href="mailto:barrykfl@gmail.com" target="_blank">barrykfl@gmail.com</a>
                          wrote:<br>
                        </div>
                        <blockquote type="cite">
                          <div dir="ltr">
                            <div>Hi :</div>
                            <div><br>
                            </div>
                            <div>What is the procedure for this minor
                              update ?</div>
                            <div><br>
                            </div>
                            <div>just yum update ipa-server after stop
                              the server?</div>
                          </div>
                        </blockquote>
                        <br>
                      </div>
                      If you just want to upgrade only the LDAP server,
                      which is the component that I for sure know is out
                      of date, then yum update 389-ds-base.<br>
                      <br>
                      Or just "yum update" - in general I don't like
                      running "franken-systems" which have a mix of
                      up-to-date and out of date packages.  Note that
                      "IPA server" is composed of several packages.<br>
                      <br>
                      You do not need to stop the server.  yum/rpm
                      upgrade will restart as needed.  If you want to
                      make sure, do ipactl restart after upgrade.
                      <div><br>
                        <br>
                        <blockquote type="cite">
                          <div dir="ltr">
                            <div>and effect of the exsitn ldap?</div>
                          </div>
                        </blockquote>
                        <br>
                      </div>
                      Not sure what you mean.  Upgrade should not touch
                      any config or data.
                      <div><br>
                        <br>
                        <blockquote type="cite">
                          <div dir="ltr">
                            <div><br>
                            </div>
                            <div>As the server 2 is master of replica
                              also , so need refo ipa-replica install ?</div>
                          </div>
                        </blockquote>
                        <br>
                      </div>
                      No, you just need to perform the same upgrade
                      procedure.
                      <div>
                        <div><br>
                          <br>
                          <blockquote type="cite">
                            <div dir="ltr">
                              <div><br>
                              </div>
                              <div>barry<br>
                              </div>
                            </div>
                            <div class="gmail_extra"><br>
                              <br>
                              <div class="gmail_quote">2014-07-09 22:20
                                GMT+08:00 Rich Megginson <span dir="ltr"><<a href="mailto:rmeggins@redhat.com" target="_blank">rmeggins@redhat.com</a>></span>:<br>
                                <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid">
                                  <div bgcolor="#FFFFFF" text="#000000">
                                    <div>
                                      <div>
                                        <div>On 07/08/2014 09:02 PM, <a href="mailto:barrykfl@gmail.com" target="_blank">barrykfl@gmail.com</a>
                                          wrote:<br>
                                        </div>
                                        <blockquote type="cite">
                                          <div dir="ltr">
                                            <div>Some error i found :</div>
                                            <div><br>
                                            </div>
                                            <div><br>
                                            </div>
                                            <div><a href="http://server1.abc.com:636" target="_blank">server1.abc.com:636</a>
(/etc/dirsrv/slapd-abc-COM)</div>
                                            <div><br>
                                            </div>
                                            <div>[29/Jun/2014:02:00:56
                                              +0800] - 389-Directory/<a href="http://1.2.11.25" target="_blank">1.2.11.25</a>
                                              B2013.325.1951 starting up</div>
                                            <div>[29/Jun/2014:02:00:56
                                              +0800] attrcrypt -
                                              attrcrypt_unwrap_key:
                                              failed to unwrap key for
                                              cipher AES</div>
                                            <div>[29/Jun/2014:02:00:56
                                              +0800] attrcrypt -
                                              attrcrypt_cipher_init:
                                              symmetric key failed to
                                              unwrap with the private
                                              key; Cert might have been
                                              renewed since the key is
                                              wrapped.  To recover the
                                              encrypted contents, keep
                                              the wrapped symmetric key
                                              value.</div>
                                            <div>[29/Jun/2014:02:00:56
                                              +0800] attrcrypt -
                                              attrcrypt_unwrap_key:
                                              failed to unwrap key for
                                              cipher 3DES</div>
                                            <div>[29/Jun/2014:02:00:56
                                              +0800] attrcrypt -
                                              attrcrypt_cipher_init:
                                              symmetric key failed to
                                              unwrap with the private
                                              key; Cert might have been
                                              renewed since the key is
                                              wrapped.  To recover the
                                              encrypted contents, keep
                                              the wrapped symmetric key
                                              value.</div>
                                            <div>[29/Jun/2014:02:00:56
                                              +0800] attrcrypt - All
                                              prepared ciphers are not
                                              available. Please disable
                                              attribute encryption.</div>
                                            <div>[29/Jun/2014:02:00:56
                                              +0800]
                                              schema-compat-plugin -
                                              warning: no entries set up
                                              under cn=computers,
                                              cn=compat,dc=abc,dc=com</div>
                                            <div>[29/Jun/2014:02:00:57
                                              +0800]
                                              schema-compat-plugin -
                                              warning: no entries set up
                                              under cn=ng,
                                              cn=compat,dc=abc,dc=com</div>
                                            <div>[29/Jun/2014:02:00:57
                                              +0800]
                                              schema-compat-plugin -
                                              warning: no entries set up
                                              under
                                              ou=sudoers,dc=abc,dc=com</div>
                                            <div>[29/Jun/2014:02:00:57
                                              +0800] - Skipping CoS
                                              Definition cn=Password
                                              Policy,cn=accounts,dc=abc,dc=com--no
                                              CoS Templates found, which
                                              should be added before the
                                              CoS Definition.</div>
                                            <div>[29/Jun/2014:02:00:57
                                              +0800] set_krb5_creds -
                                              Could not get initial
                                              credentials for principal
                                              [<a href="mailto:ldap/server1.abc.com@abc.COM" target="_blank">ldap/server1.abc.com@abc.COM</a>]
                                              in keytab [<a>FILE:/etc/dirsrv/ds.keytab</a>]:


                                              -1765328228 (Cannot
                                              contact any KDC for
                                              requested realm)</div>
                                            <div>[29/Jun/2014:02:00:58
                                              +0800] - Skipping CoS
                                              Definition cn=Password
                                              Policy,cn=accounts,dc=abc,dc=com--no
                                              CoS Templates found, which
                                              should be added before the
                                              CoS Definition.</div>
                                            <div>[29/Jun/2014:02:00:58
                                              +0800]
                                              slapd_ldap_sasl_interactive_bind
                                              - Error: could not perform
                                              interactive bind for id []
                                              mech [GSSAPI]: LDAP error
                                              -2 (Local error)
                                              (SASL(-1): generic
                                              failure: GSSAPI Error:
                                              Unspecified GSS failure.
                                               Minor code may provide
                                              more information
                                              (Credentials cache file
                                              '/tmp/krb5cc_492' not
                                              found)) errno 0 (Success)</div>
                                            <div>[29/Jun/2014:02:00:58
                                              +0800] slapi_ldap_bind -
                                              Error: could not perform
                                              interactive bind for id []
                                              mech [GSSAPI]: error -2
                                              (Local error)</div>
                                            <div>[29/Jun/2014:02:00:58
                                              +0800]
                                              NSMMReplicationPlugin -
                                              agmt="cn=<a href="http://meToserver2.abc.com" target="_blank">meToserver2.abc.com</a>"
                                              (server2:389): Replication
                                              bind with GSSAPI auth
                                              failed: LDAP error -2
                                              (Local error) (SASL(-1):
                                              generic failure: GSSAPI
                                              Error: Unspecified GSS
                                              failure.  Minor code may
                                              provide more information
                                              (Credentials cache file
                                              '/tmp/krb5cc_492' not
                                              found))</div>
                                            <div>[29/Jun/2014:02:00:58
                                              +0800] - slapd started.
                                               Listening on All
                                              Interfaces port 389 for
                                              LDAP requests</div>
                                            <div>[29/Jun/2014:02:00:58
                                              +0800] - Listening on All
                                              Interfaces port 636 for
                                              LDAPS requests</div>
                                            <div><br>
                                            </div>
                                            <div><br>
                                            </div>
                                            <div>
                                              <div>389-Directory/<a href="http://1.2.11.15" target="_blank">1.2.11.15</a> B2013.240.174</div>
                                              <div><a href="http://server2.abc.com:636" target="_blank">server2.abc.com:636</a>
(/etc/dirsrv/slapd-abc-COM)</div>
                                              <div><br>
                                              </div>
                                              <div>[30/Jun/2014:12:51:31
                                                +0800]
                                                slapd_ldap_sasl_interactive_bind
                                                - Error: could not
                                                perform interactive bind
                                                for id [] mech [GSSAPI]:
                                                LDAP error -2 (Local
                                                error) (SASL(-1):
                                                generic failure: GSSAPI
                                                Error: Unspecified GSS
                                                failure.  Minor code may
                                                provide more information
                                                (Ticket expired)) errno
                                                0 (Success)</div>
                                              <div>[30/Jun/2014:12:51:31
                                                +0800]
                                                slapd_ldap_sasl_interactive_bind
                                                - Error: could not
                                                perform interactive bind
                                                for id [] mech [GSSAPI]:
                                                LDAP error -2 (Local
                                                error) (SASL(-1):
                                                generic failure: GSSAPI
                                                Error: Unspecified GSS
                                                failure.  Minor code may
                                                provide more information
                                                (Ticket expired)) errno
                                                0 (Success)</div>
                                              <div>[30/Jun/2014:12:51:31
                                                +0800] slapi_ldap_bind -
                                                Error: could not perform
                                                interactive bind for id
                                                [] mech [GSSAPI]: error
                                                -2 (Local error)</div>
                                              <div>[30/Jun/2014:12:51:31
                                                +0800]
                                                NSMMReplicationPlugin -
                                                agmt="cn=<a href="http://meToserver1.abc.com" target="_blank">meToserver1.abc.com</a>"
                                                (server1:389):
                                                Replication bind with
                                                GSSAPI auth failed: LDAP
                                                error -2 (Local error)
                                                (SASL(-1): generic
                                                failure: GSSAPI Error:
                                                Unspecified GSS failure.
                                                 Minor code may provide
                                                more information (Ticket
                                                expired))</div>
                                              <div>[30/Jun/2014:12:51:34
                                                +0800]
                                                slapd_ldap_sasl_interactive_bind
                                                - Error: could not
                                                perform interactive bind
                                                for id [] mech [GSSAPI]:
                                                LDAP error -2 (Local
                                                error) (SASL(-1):
                                                generic failure: GSSAPI
                                                Error: Unspecified GSS
                                                failure.  Minor code may
                                                provide more information
                                                (Ticket expired)) errno
                                                0 (Success)</div>
                                              <div>[30/Jun/2014:12:51:35
                                                +0800]
                                                slapd_ldap_sasl_interactive_bind
                                                - Error: could not
                                                perform interactive bind
                                                for id [] mech [GSSAPI]:
                                                LDAP error -2 (Local
                                                error) (SASL(-1):
                                                generic failure: GSSAPI
                                                Error: Unspecified GSS
                                                failure.  Minor code may
                                                provide more information
                                                (Ticket expired)) errno
                                                0 (Success)</div>
                                              <div>[30/Jun/2014:12:51:35
                                                +0800] slapi_ldap_bind -
                                                Error: could not perform
                                                interactive bind for id
                                                [] mech [GSSAPI]: error
                                                -2 (Local error)</div>
                                              <div>[30/Jun/2014:12:51:40
                                                +0800]
                                                slapd_ldap_sasl_interactive_bind
                                                - Error: could not
                                                perform interactive bind
                                                for id [] mech [GSSAPI]:
                                                LDAP error -2 (Local
                                                error) (SASL(-1):
                                                generic failure: GSSAPI
                                                Error: Unspecified GSS
                                                failure.  Minor code may
                                                provide more information
                                                (Ticket expired)) errno
                                                0 (Success)</div>
                                              <div>[30/Jun/2014:12:51:40
                                                +0800]
                                                slapd_ldap_sasl_interactive_bind
                                                - Error: could not
                                                perform interactive bind
                                                for id [] mech [GSSAPI]:
                                                LDAP error -2 (Local
                                                error) (SASL(-1):
                                                generic failure: GSSAPI
                                                Error: Unspecified GSS
                                                failure.  Minor code may
                                                provide more information
                                                (Ticket expired)) errno
                                                0 (Success)</div>
                                              <div>[30/Jun/2014:12:51:40
                                                +0800] slapi_ldap_bind -
                                                Error: could not perform
                                                interactive bind for id
                                                [] mech [GSSAPI]: error
                                                -2 (Local error)</div>
                                              <div>[30/Jun/2014:12:51:52
                                                +0800]
                                                NSMMReplicationPlugin -
                                                agmt="cn=<a href="http://meToserver1.abc.com" target="_blank">meToserver1.abc.com</a>"
                                                (server1:389):
                                                Replication bind with
                                                GSSAPI auth resumed</div>
                                            </div>
                                          </div>
                                          <div class="gmail_extra"><br>
                                          </div>
                                        </blockquote>
                                        <br>
                                      </div>
                                    </div>
                                    You are using an older version of
                                    389.  The version on server2 is
                                    older than the version on server1. 
                                    Can you upgrade and see if that
                                    fixes your problems?  Even if it
                                    doesn't fix your problems, it will
                                    be much easier for us to support.
                                    <div>
                                      <div><br>
                                        <br>
                                        <blockquote type="cite">
                                          <div class="gmail_extra"><br>
                                            <div class="gmail_quote">2014-07-09

                                              10:55 GMT+08:00 <span dir="ltr"><<a href="mailto:barrykfl@gmail.com" target="_blank">barrykfl@gmail.com</a>></span>:<br>
                                              <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid">
                                                <div dir="ltr">
                                                  <div>FYI..</div>
                                                  <div>160:
                                                    [04/Jul/2014:12:35:30
                                                    +0800] conn=936207
                                                    fd=73 slot=73
                                                    connection from
                                                    192.168.156.89 to
                                                    192.168.156.89</div>
                                                  <div>163:
                                                    [04/Jul/2014:12:35:30
                                                    +0800] conn=936207
                                                    op=-1 fd=73 closed -
                                                    B1</div>
                                                  <div><br>
                                                  </div>
                                                  <div>There is not abt
                                                    binding but i unsure
                                                    how to fix ..</div>
                                                  <div><br>
                                                  </div>
                                                  <div><br>
                                                  </div>
                                                </div>
                                                <div class="gmail_extra"><br>
                                                  <br>
                                                  <div class="gmail_quote">2014-07-09

                                                    2:01 GMT+08:00 Rich
                                                    Megginson <span dir="ltr"><<a href="mailto:rmeggins@redhat.com" target="_blank">rmeggins@redhat.com</a>></span>:
                                                    <div>
                                                      <div><br>
                                                        <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid">
                                                          <div bgcolor="#FFFFFF" text="#000000">
                                                          <div>
                                                          <div>On
                                                          07/08/2014
                                                          02:16 AM, <a href="mailto:barrykfl@gmail.com" target="_blank">barrykfl@gmail.com</a>
                                                          wrote:<br>
                                                          </div>
                                                          <blockquote type="cite">
                                                          <div dir="ltr">Resent

                                                          as size limit.
                                                          <div><br>
                                                          <div><br>
                                                          </div>
                                                          <div>
                                                          <div style="font-family:arial,sans-serif;font-size:14px">Here




                                                          u are  server1
                                                          's access log
                                                          seem one side
                                                          broken<br>
                                                          </div>
                                                          <div style="font-family:arial,sans-serif;font-size:14px">
                                                          <br>
                                                          </div>
                                                          <div style="font-family:arial,sans-serif;font-size:14px">the




                                                          problem is how
                                                          to make it
                                                          replicate
                                                          again.</div>
                                                          <div style="font-family:arial,sans-serif;font-size:14px"><br>
                                                          </div>
                                                          <div style="font-family:arial,sans-serif;font-size:14px">
                                                          At server 1</div>
                                                          <div style="font-family:arial,sans-serif;font-size:14px">
                                                          <div><br>
                                                          </div>
                                                          <div>it is ok
                                                           master
                                                          server1 master
                                                          server2 <br>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          </div>
                                                          <div style="font-family:arial,sans-serif;font-size:14px">
                                                          Another side
                                                          server 2
                                                          contains 2 ip
                                                          replication.</div>
                                                          <div style="font-family:arial,sans-serif;font-size:14px">
                                                          <div><br>
                                                          </div>
                                                          <div>ipa-replica-manage


                                                          list shown
                                                          Can't contact
                                                          LDAP server<br>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          </div>
                                                          <div style="font-family:arial,sans-serif;font-size:14px">I
                                                          dont know why
                                                          but the
                                                          prolematic
                                                          server is
                                                          sever 2 not
                                                          server 1</div>
                                                          <div style="font-family:arial,sans-serif;font-size:14px"><br>
                                                          </div>
                                                          <div style="font-family:arial,sans-serif;font-size:14px">
                                                          log of server2</div>
                                                          <div style="font-family:arial,sans-serif;font-size:14px">[08/Jul/2014:16:02:40




                                                          +0800]
                                                          conn=3299731
                                                          fd=69 slot=69
                                                          connection
                                                          from
                                                          192.168.15.89
                                                          (server1) to
                                                          192.168.15.88(server2)<br>
                                                          </div>
                                                          <div style="font-family:arial,sans-serif;font-size:14px">
                                                          <div>[08/Jul/2014:16:02:40


                                                          +0800]
                                                          conn=3299731
                                                          op=-1 fd=69
                                                          closed - B1</div>
                                                          <div>[08/Jul/2014:16:02:40


                                                          +0800]
                                                          conn=3299732
                                                          fd=69 slot=69
                                                          connection
                                                          from
                                                          192.168.15.89
                                                          to
                                                          192.168.15.88</div>
                                                          <div>[08/Jul/2014:16:02:40


                                                          +0800]
                                                          conn=3299732
                                                          op=-1 fd=69
                                                          closed - B1</div>
                                                          <div>[08/Jul/2014:16:02:41


                                                          +0800]
                                                          conn=3299733
                                                          fd=69 slot=69
                                                          connection
                                                          from
                                                          192.168.15.89
                                                          to
                                                          192.168.15.88</div>
                                                          <div>[08/Jul/2014:16:02:41


                                                          +0800]
                                                          conn=3299733
                                                          op=-1 fd=69
                                                          closed - B1</div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          <br>
                                                          </div>
                                                          You never
                                                          answered my
                                                          question
                                                          below.  "Are
                                                          you sure that
                                                          this
                                                          connection is
                                                          a replication
                                                          session?  Can
                                                          you post all
                                                          of the
                                                          operations
                                                          from the
                                                          access log
                                                          from
                                                          conn=936207?"<br>
                                                          <br>
                                                          In the future,
                                                          please avoid
                                                          spamming the
                                                          list with
                                                          large log
                                                          files.  In
                                                          general, it's
                                                          better to
                                                          provide
                                                          excerpts from
                                                          the log files
                                                          showing the
                                                          problem, paste
                                                          them to <a href="http://fpaste.org" target="_blank">fpaste.org</a>, and post the
                                                          link to the
                                                          mailing list. 
                                                          If for some
                                                          reason you
                                                          need to post a
                                                          large file,
                                                          please use a
                                                          file sharing
                                                          service and
                                                          post the link
                                                          to the file.<br>
                                                          <br>
                                                          Can you take a
                                                          look at your
                                                          errors log
                                                          from server 1
                                                          and server 2
                                                          and see if
                                                          there are any
                                                          relevant
                                                          errors?<br>
                                                          <br>
                                                          If I had to
                                                          guess, I would
                                                          say that there
                                                          is some sort
                                                          of network
                                                          error between
                                                          server 1 and
                                                          server 2 that
                                                          causes the
                                                          excessive
                                                          closed - B1. 
                                                          Perhaps there
                                                          will be more
                                                          information in
                                                          the errors
                                                          log.
                                                          <div>
                                                          <div><br>
                                                          <br>
                                                          <blockquote type="cite">
                                                          <div dir="ltr">
                                                          <div>
                                                          <div>
                                                          <div style="font-family:arial,sans-serif;font-size:14px">
                                                          <div><br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          <div class="gmail_extra"><br>
                                                          <br>
                                                          <div class="gmail_quote">2014-07-07



                                                          22:21
                                                          GMT+08:00 Rich
                                                          Megginson <span dir="ltr"><<a href="mailto:rmeggins@redhat.com" target="_blank">rmeggins@redhat.com</a>></span>:<br>
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid">



                                                          <div bgcolor="#FFFFFF" text="#000000">
                                                          <div>
                                                          <div>On
                                                          07/04/2014
                                                          03:28 AM, <a href="mailto:barrykfl@gmail.com" target="_blank">barrykfl@gmail.com</a>
                                                          wrote:<br>
                                                          </div>
                                                          <blockquote type="cite">
                                                          <div dir="ltr">FOUND


                                                          something
                                                          strange that
                                                          server 1
                                                          replicate to
                                                          itself rather
                                                          than server2
                                                          <div><br>
                                                          </div>
                                                          <div>
                                                          <div>Server1
                                                          access log
                                                          > Wrong</div>
                                                          <div>[04/Jul/2014:12:35:30



                                                          +0800]
                                                          conn=936207
                                                          fd=73 slot=73
                                                          connection
                                                          from
                                                          192.168.15.89(
                                                          server1 )  to
                                                          192.168.15.89
                                                          (server1)</div>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          <br>
                                                          </div>
                                                          Are you sure
                                                          that this
                                                          connection is
                                                          a replication
                                                          session?  Can
                                                          you post all
                                                          of the
                                                          operations
                                                          from the
                                                          access log
                                                          from
                                                          conn=936207?
                                                          <div>
                                                          <div><br>
                                                          <br>
                                                          <blockquote type="cite">
                                                          <div dir="ltr">
                                                          <div>
                                                          <div><br>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          <div>Server 2
                                                          access log
                                                          > OK</div>
                                                          <div>[04/Jul/2014:12:35:30



                                                          +0800]
                                                          conn=936208
                                                          fd=74 slot=74
                                                          connection
                                                          from
                                                          192.168.15.89(server2)
                                                          to
                                                          192.168.15.88
                                                          (server2)</div>
                                                          </div>
                                                          </div>
                                                          <div class="gmail_extra">
                                                          <br>
                                                          <br>
                                                          <div class="gmail_quote">2014-07-04



                                                          9:25 GMT+08:00
                                                          <span dir="ltr"><<a href="mailto:barrykfl@gmail.com" target="_blank">barrykfl@gmail.com</a>></span>:<br>
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid">



                                                          <div dir="ltr">
                                                          <div>Just sure
                                                          now one side
                                                          flow is
                                                          broken, if u
                                                          update server1
                                                          , it 100% work
                                                          server2 will
                                                          upgrade.<br>
                                                          </div>
                                                          <div>but if u
                                                          update server2
                                                          there is
                                                          chance non-syn
                                                          e.g it create
                                                          username  in
                                                          server1 with
                                                          posfix grp
                                                          >ok</div>
                                                          <div>but in
                                                          server2 it
                                                          only created
                                                          posfix grp but
                                                          no username
                                                          /attribute it
                                                          occur serveral
                                                          times. I have
                                                          to use command
                                                          line grp del
                                                          ...etc. to
                                                          force del them
                                                          and recreate
                                                          them.,.</div>
                                                          <div><br>
                                                          </div>
                                                          <div>Result
                                                          below:</div>
                                                          <div><br>
                                                          </div>
                                                          <div><a href="http://server2.abc.com" target="_blank">server2.abc.com</a>:
                                                          replica</div>
                                                          <div>  last
                                                          init status:
                                                          None</div>
                                                          <div>  last
                                                          init ended:
                                                          None</div>
                                                          <div>  last
                                                          update status:
                                                          0 Replica
                                                          acquired
                                                          successfully:
                                                          Incremental
                                                          update
                                                          succeeded</div>
                                                          <div>  last
                                                          update ended:
                                                          2014-07-04
                                                          00:33:18+00:00</div>
                                                          <div><br>
                                                          </div>
                                                          <div>Directory
                                                          Manager
                                                          password:</div>
                                                          <div><br>
                                                          </div>
                                                          <div><a href="http://server1.abc.com" target="_blank">server1.abc.com</a>:
                                                          replica</div>
                                                          <div>  last
                                                          init status: 0
                                                          Total update
                                                          succeeded</div>
                                                          <div>  last
                                                          init ended:
                                                          2014-06-20
                                                          10:07:02+00:00</div>
                                                          <div>  last
                                                          update status:
                                                          0 Replica
                                                          acquired
                                                          successfully:
                                                          Incremental
                                                          update
                                                          succeeded</div>
                                                          <div>  last
                                                          update ended:
                                                          2014-07-04
                                                          01:14:19+00:00</div>
                                                          <div><br>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          <div>[root@(LIVE)server2



                                                          ~]$  ipactl
                                                          status</div>
                                                          <div>Directory
                                                          Service:
                                                          RUNNING</div>
                                                          <div>KDC
                                                          Service:
                                                          RUNNING</div>
                                                          <div>KPASSWD
                                                          Service:
                                                          RUNNING</div>
                                                          <div>MEMCACHE
                                                          Service:
                                                          RUNNING</div>
                                                          <div> HTTP
                                                          Service:
                                                          RUNNING</div>
                                                          </div>
                                                          <div class="gmail_extra"><br>
                                                          <br>
                                                          <div class="gmail_quote">2014-07-04



                                                          1:34 GMT+08:00
                                                          Rob Crittenden
                                                          <span dir="ltr"><<a href="mailto:rcritten@redhat.com" target="_blank">rcritten@redhat.com</a>></span>:
                                                          <div>
                                                          <div><br>
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid">



                                                          <div><a href="mailto:barrykfl@gmail.com" target="_blank">barrykfl@gmail.com</a>
                                                          wrote:<br>
                                                          > Yes they
                                                          are running.
                                                          Server 1 can
                                                          syn to server2
                                                          but error at
                                                          server 2<br>
                                                          > like
                                                          this.<br>
                                                          <br>
                                                          </div>
                                                          How do you
                                                          know server 1
                                                          is syncing
                                                          with server 2?<br>
                                                          <br>
                                                          On server 1
                                                          I'd run:<br>
                                                          <br>
                                                          ipa-replica-manage



                                                          list -v
                                                          `hostname`<br>
                                                          <br>
                                                          This will show
                                                          the
                                                          replication
                                                          status.<br>
                                                          <br>
                                                          And what does
                                                          ipactl status
                                                          show on server
                                                          2?<br>
                                                          <br>
                                                          rob<br>
                                                          <div><br>
                                                          ><br>
                                                          > 2014/7/3
                                                          下午10:14 於 "Rob
                                                          Crittenden"
                                                          <<a href="mailto:rcritten@redhat.com" target="_blank">rcritten@redhat.com</a><br>
                                                          </div>
                                                          >
                                                          <mailto:<a href="mailto:rcritten@redhat.com" target="_blank">rcritten@redhat.com</a>>>





                                                          寫道:<br>
                                                          <div>><br>
                                                          >    
                                                          Please keep
                                                          relies on the
                                                          list.<br>
                                                          ><br>
                                                          </div>
                                                          <div>>    
                                                          <a href="mailto:barrykfl@gmail.com" target="_blank">barrykfl@gmail.com</a>
                                                          <mailto:<a href="mailto:barrykfl@gmail.com" target="_blank">barrykfl@gmail.com</a>>





                                                          wrote:<br>
                                                          >     >
                                                          I saw the
                                                          error beloe
                                                          and errpr log
                                                          is it related
                                                          ?<br>
                                                          >     ><br>
                                                          >     >
                                                          29/Jun/2014:02:00:58




                                                          +0800]
                                                          slapd_ldap_sasl_interactive_bind
                                                          - Error:<br>
                                                          >     >
                                                          could not
                                                          perform
                                                          interactive
                                                          bind for id []
                                                          mech [GSSAPI]:
                                                          LDAP error<br>
                                                          >     >
                                                          -2 (Local
                                                          error)
                                                          (SASL(-1):
                                                          generic
                                                          failure:
                                                          GSSAPI Error:
                                                          Unspecified<br>
                                                          >     >
                                                          GSS failure.
                                                           Minor code
                                                          may provide
                                                          more
                                                          information
                                                          (Credentials<br>
                                                          >     cache<br>
                                                          >     >
                                                          file
                                                          '/tmp/krb5cc_492'
                                                          not found))
                                                          errno 0
                                                          (Success)<br>
                                                          >     >
                                                          [29/Jun/2014:02:00:58



                                                          +0800]
                                                          slapi_ldap_bind
                                                          - Error: could
                                                          not<br>
                                                          >    
                                                          perform<br>
                                                          >     >
                                                          interactive
                                                          bind for id []
                                                          mech [GSSAPI]:
                                                          error -2
                                                          (Local error)<br>
                                                          ><br>
                                                          >     I
                                                          believe this
                                                          is fairly
                                                          normal on a
                                                          new startup.
                                                          It has to
                                                          start<br>
                                                          >    
                                                          somewhere. The
                                                          expired ticket
                                                          errors below
                                                          are unexpected
                                                          since there<br>
                                                          >     are
                                                          so many of
                                                          them. Is your
                                                          KDC running?<br>
                                                          ><br>
                                                          >    
                                                          ipactl status<br>
                                                          ><br>
                                                          >     rob<br>
                                                          ><br>
                                                          >     ><br>
                                                          >     ><br>
                                                          >     >
                                                          2014-07-02
                                                          14:15
                                                          GMT+08:00 <<a href="mailto:barrykfl@gmail.com" target="_blank">barrykfl@gmail.com</a><br>
                                                          </div>
                                                          >    
                                                          <mailto:<a href="mailto:barrykfl@gmail.com" target="_blank">barrykfl@gmail.com</a>>





                                                          <mailto:<a href="mailto:barrykfl@gmail.com" target="_blank">barrykfl@gmail.com</a><br>
                                                          <div>
                                                          <div>>    
                                                          <mailto:<a href="mailto:barrykfl@gmail.com" target="_blank">barrykfl@gmail.com</a>>>>:<br>
                                                          >     ><br>
                                                          >     ><br>
                                                          >     >
                                                              this is
                                                          the error log
                                                          i found at <a href="http://2.abc.com" target="_blank">2.abc.com</a>
                                                          <<a href="http://2.abc.com" target="_blank">http://2.abc.com</a>><br>
                                                          >     <<a href="http://2.abc.com" target="_blank">http://2.abc.com</a>><br>
                                                          >     ><br>
                                                          >     >
                                                             
                                                          [30/Jun/2014:12:51:31
                                                          +0800]
                                                          slapd_ldap_sasl_interactive_bind
                                                          -<br>
                                                          >     >
                                                              Error:
                                                          could not
                                                          perform
                                                          interactive
                                                          bind for id []
                                                          mech [GSSAPI]:<br>
                                                          >     >
                                                              LDAP error
                                                          -2 (Local
                                                          error)
                                                          (SASL(-1):
                                                          generic
                                                          failure:
                                                          GSSAPI<br>
                                                          >     >
                                                              Error:
                                                          Unspecified
                                                          GSS failure.
                                                           Minor code
                                                          may provide
                                                          more<br>
                                                          >     >
                                                             
                                                          information
                                                          (Ticket
                                                          expired))
                                                          errno 0
                                                          (Success)<br>
                                                          >     >
                                                             
                                                          [30/Jun/2014:12:51:31
                                                          +0800]
                                                          slapd_ldap_sasl_interactive_bind
                                                          -<br>
                                                          >     >
                                                              Error:
                                                          could not
                                                          perform
                                                          interactive
                                                          bind for id []
                                                          mech [GSSAPI]:<br>
                                                          >     >
                                                              LDAP error
                                                          -2 (Local
                                                          error)
                                                          (SASL(-1):
                                                          generic
                                                          failure:
                                                          GSSAPI<br>
                                                          >     >
                                                              Error:
                                                          Unspecified
                                                          GSS failure.
                                                           Minor code
                                                          may provide
                                                          more<br>
                                                          >     >
                                                             
                                                          information
                                                          (Ticket
                                                          expired))
                                                          errno 0
                                                          (Success)<br>
                                                          >     >
                                                             
                                                          [30/Jun/2014:12:51:31
                                                          +0800]
                                                          slapi_ldap_bind
                                                          - Error: could
                                                          not<br>
                                                          >     >
                                                              perform
                                                          interactive
                                                          bind for id []
                                                          mech [GSSAPI]:
                                                          error -2<br>
                                                          >    
                                                          (Local error)<br>
                                                          >     >
                                                             
                                                          [30/Jun/2014:12:51:31
                                                          +0800]
                                                          NSMMReplicationPlugin
                                                          -<br>
                                                          >     >
                                                              agmt="cn=<a href="http://meTo1.abc.com" target="_blank">meTo1.abc.com</a>
                                                          <<a href="http://meTo1.abc.com" target="_blank">http://meTo1.abc.com</a>><br>
                                                          >     <<a href="http://meTo1.abc.com" target="_blank">http://meTo1.abc.com</a>>"





                                                          (central:389):<br>
                                                          >     >
                                                             
                                                          Replication
                                                          bind with
                                                          GSSAPI auth
                                                          failed: LDAP
                                                          error -2
                                                          (Local<br>
                                                          >     >
                                                              error)
                                                          (SASL(-1):
                                                          generic
                                                          failure:
                                                          GSSAPI Error:
                                                          Unspecified
                                                          GSS<br>
                                                          >     >
                                                              failure.
                                                           Minor code
                                                          may provide
                                                          more
                                                          information
                                                          (Ticket<br>
                                                          >    
                                                          expired))<br>
                                                          >     >
                                                             
                                                          [30/Jun/2014:12:51:34
                                                          +0800]
                                                          slapd_ldap_sasl_interactive_bind
                                                          -<br>
                                                          >     >
                                                              Error:
                                                          could not
                                                          perform
                                                          interactive
                                                          bind for id []
                                                          mech [GSSAPI]:<br>
                                                          >     >
                                                              LDAP error
                                                          -2 (Local
                                                          error)
                                                          (SASL(-1):
                                                          generic
                                                          failure:
                                                          GSSAPI<br>
                                                          >     >
                                                              Error:
                                                          Unspecified
                                                          GSS failure.
                                                           Minor code
                                                          may provide
                                                          more<br>
                                                          >     >
                                                             
                                                          information
                                                          (Ticket
                                                          expired))
                                                          errno 0
                                                          (Success)<br>
                                                          >     >
                                                             
                                                          [30/Jun/2014:12:51:35
                                                          +0800]
                                                          slapd_ldap_sasl_interactive_bind
                                                          -<br>
                                                          >     >
                                                              Error:
                                                          could not
                                                          perform
                                                          interactive
                                                          bind for id []
                                                          mech [GSSAPI]:<br>
                                                          >     >
                                                              LDAP error
                                                          -2 (Local
                                                          error)
                                                          (SASL(-1):
                                                          generic
                                                          failure:
                                                          GSSAPI<br>
                                                          >     >
                                                              Error:
                                                          Unspecified
                                                          GSS failure.
                                                           Minor code
                                                          may provide
                                                          more<br>
                                                          >     >
                                                             
                                                          information
                                                          (Ticket
                                                          expired))
                                                          errno 0
                                                          (Success)<br>
                                                          >     >
                                                             
                                                          [30/Jun/2014:12:51:35
                                                          +0800]
                                                          slapi_ldap_bind
                                                          - Error: could
                                                          not<br>
                                                          >     >
                                                              perform
                                                          interactive
                                                          bind for id []
                                                          mech [GSSAPI]:
                                                          error -2<br>
                                                          >    
                                                          (Local error)<br>
                                                          >     >
                                                             
                                                          [30/Jun/2014:12:51:40
                                                          +0800]
                                                          slapd_ldap_sasl_interactive_bind
                                                          -<br>
                                                          >     >
                                                              Error:
                                                          could not
                                                          perform
                                                          interactive
                                                          bind for id []
                                                          mech [GSSAPI]:<br>
                                                          >     >
                                                              LDAP error
                                                          -2 (Local
                                                          error)
                                                          (SASL(-1):
                                                          generic
                                                          failure:
                                                          GSSAPI<br>
                                                          >     >
                                                              Error:
                                                          Unspecified
                                                          GSS failure.
                                                           Minor code
                                                          may provide
                                                          more<br>
                                                          >     >
                                                             
                                                          information
                                                          (Ticket
                                                          expired))
                                                          errno 0
                                                          (Success)<br>
                                                          >     >
                                                             
                                                          [30/Jun/2014:12:51:40
                                                          +0800]
                                                          slapd_ldap_sasl_interactive_bind
                                                          -<br>
                                                          >     >
                                                              Error:
                                                          could not
                                                          perform
                                                          interactive
                                                          bind for id []
                                                          mech [GSSAPI]:<br>
                                                          >     >
                                                              LDAP error
                                                          -2 (Local
                                                          error)
                                                          (SASL(-1):
                                                          generic
                                                          failure:
                                                          GSSAPI<br>
                                                          >     >
                                                              Error:
                                                          Unspecified
                                                          GSS failure.
                                                           Minor code
                                                          may provide
                                                          more<br>
                                                          >     >
                                                             
                                                          information
                                                          (Ticket
                                                          expired))
                                                          errno 0
                                                          (Success)<br>
                                                          >     >
                                                             
                                                          [30/Jun/2014:12:51:40
                                                          +0800]
                                                          slapi_ldap_bind
                                                          - Error: could
                                                          not<br>
                                                          >     >
                                                              perform
                                                          interactive
                                                          bind for id []
                                                          mech [GSSAPI]:
                                                          error -2<br>
                                                          >    
                                                          (Local error)<br>
                                                          >     ><br>
                                                          >     ><br>
                                                          >     >
                                                              2014-07-02
                                                          12:32
                                                          GMT+08:00 <<a href="mailto:barrykfl@gmail.com" target="_blank">barrykfl@gmail.com</a><br>
                                                          >    
                                                          <mailto:<a href="mailto:barrykfl@gmail.com" target="_blank">barrykfl@gmail.com</a>><br>
                                                          </div>
                                                          </div>
                                                          >     >
                                                             
                                                          <mailto:<a href="mailto:barrykfl@gmail.com" target="_blank">barrykfl@gmail.com</a>
                                                          <mailto:<a href="mailto:barrykfl@gmail.com" target="_blank">barrykfl@gmail.com</a>>>>:<br>
                                                          <div>>    
                                                          ><br>
                                                          >     >
                                                                  yes on
                                                          node 1 it is
                                                          happening only
                                                          node2 fail
                                                          connect<br>
                                                          >     ><br>
                                                          >     >
                                                                 
                                                          ipa-replica-manage
                                                          list <a href="http://2.abc.com" target="_blank">2.abc.com</a> <<a href="http://2.abc.com" target="_blank">http://2.abc.com</a>><br>
                                                          >     <<a href="http://2.abc.com" target="_blank">http://2.abc.com</a>><br>
                                                          >     >
                                                                 
                                                          Directory
                                                          Manager
                                                          password:<br>
                                                          >     ><br>
                                                          </div>
                                                          >     >
                                                                  <a href="http://1.abc.com" target="_blank">1.abc.com</a> <<a href="http://1.abc.com" target="_blank">http://1.abc.com</a>> <<a href="http://1.abc.com" target="_blank">http://1.abc.com</a>>:





                                                          replica<br>
                                                          <div>>    
                                                          ><br>
                                                          >     ><br>
                                                          >     ><br>
                                                          >     >
                                                                 
                                                          2014-06-30
                                                          20:59
                                                          GMT+08:00 Rob
                                                          Crittenden<br>
                                                          >     <<a href="mailto:rcritten@redhat.com" target="_blank">rcritten@redhat.com</a>
                                                          <mailto:<a href="mailto:rcritten@redhat.com" target="_blank">rcritten@redhat.com</a>><br>
                                                          </div>
                                                          >     >
                                                                 
                                                          <mailto:<a href="mailto:rcritten@redhat.com" target="_blank">rcritten@redhat.com</a>
                                                          <mailto:<a href="mailto:rcritten@redhat.com" target="_blank">rcritten@redhat.com</a>>>>:<br>
                                                          <div>
                                                          <div>>    
                                                          ><br>
                                                          >     >
                                                                     
                                                          Barry wrote:<br>
                                                          >     >
                                                                     
                                                          > Hi:<br>
                                                          >     >
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          </div>
                                                          </div>
                                                        </blockquote>
                                                      </div>
                                                    </div>
                                                  </div>
                                                </div>
                                              </blockquote>
                                            </div>
                                          </div>
                                        </blockquote>
                                      </div>
                                    </div>
                                  </div>
                                </blockquote>
                              </div>
                            </div>
                          </blockquote>
                        </div>
                      </div>
                    </div>
                  </blockquote>
                </div>
              </div>
            </blockquote>
          </div>
          ...</blockquote>
      </div>
    </blockquote>
    <br>
  </div>

</blockquote></div><br></div>
</blockquote></div><br></div>
</blockquote></div><br></div>