[Freeipa-users] ipa-replica-manage list fail on server 2

Rich Megginson rmeggins at redhat.com
Mon Jul 14 16:03:39 UTC 2014


On 07/13/2014 08:51 PM, barrykfl at gmail.com wrote:
> Hi:
>
> Only for the servers that are getting the "DB_LOCK_DEADLOCK: Locker 
> killed to resolve a deadlock" message in the errors log.
>
> > need restart ipactl service after modifcation?
>
> But this does not explain the "cant contact ldap server" errors.
>
> Which ipa commands give the "cant contact ldap server" errors?
>
> > server2.abc.com <http://server2.abc.com>  and command related ipa 
> shown can't contact ldap sver , log shown before.

Does this mean that
ipa user-find
on server2.abc.com gives a "cant contact ldap server" error?

Or is it only the ipa replica-manage status command that gives this error?

If it is the former, does ldapsearch work?  Does kinit work?

>
>
> 2014-07-11 21:55 GMT+08:00 Rich Megginson <rmeggins at redhat.com 
> <mailto:rmeggins at redhat.com>>:
>
>     On 07/11/2014 01:53 AM, barrykfl at gmail.com
>     <mailto:barrykfl at gmail.com> wrote:
>>     At server 2 there is a error:
>>
>>
>>     [10/Jul/2014:12:29:59 +0800] NSMMReplicationPlugin -
>>     agmt="cn=meToserver1.abc.com <http://meToserver1.abc.com>"
>>     (central:389): Replication bind with GSSAPI auth failed: LDAP
>>     error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error:
>>     Unspecified GSS failure.  Minor code may provide more information
>>     (Credentials cache file '/tmp/krb5cc_494' not found))
>
>     This is usually a transient error that should go away.
>
>>
>>
>>     2014-07-11 10:26 GMT+08:00 <barrykfl at gmail.com
>>     <mailto:barrykfl at gmail.com>>:
>>
>>         Yes ,
>>         still get "cant contact ldap server" after upgrading both
>>         servers.
>>
>>
>>         2014-07-10 23:18 GMT+08:00 Rich Megginson
>>         <rmeggins at redhat.com <mailto:rmeggins at redhat.com>>:
>>
>>             On 07/10/2014 09:15 AM, barrykfl at gmail.com
>>             <mailto:barrykfl at gmail.com> wrote:
>>>
>>>             But any hint that server 2 say cant contact ldap server
>>>             if type ipa command?
>>>
>>
>>             Please keep replies on list.
>>
>>             You still get "cant contact ldap server" after upgrading
>>             both servers?
>>
>>>             2014/7/10 下午10:25 於 "Rich Megginson"
>>>             <rmeggins at redhat.com <mailto:rmeggins at redhat.com>> 寫道:
>>>
>>>                 On 07/10/2014 01:14 AM, barrykfl at gmail.com
>>>                 <mailto:barrykfl at gmail.com> wrote:
>>>>                 Tried and now two version same ....but seem same
>>>>                 situation.
>>>>
>>>>                 i found a related error log that server1 has
>>>>                 account after added user but not replicated to
>>>>                 server2. Is it too fast on UI clicking ? as i exp
>>>>                 once that click very
>>>>                 fast twice add and edit user may cause server 2 no
>>>>                 record.
>>>>
>>>>
>>>>                 [10/Jul/2014:14:20:01 +0800] NSMMReplicationPlugin
>>>>                 - changelog program - _cl5WriteOperationTxn: retry
>>>>                 (49) the transaction (csn=53be3097000000040000)
>>>>                 failed (rc=-30994 (DB_LOCK_DEADLOCK: Locker killed
>>>>                 to resolve a deadlock))
>>>>                 [10/Jul/2014:14:20:01 +0800] NSMMReplicationPlugin
>>>>                 - changelog program - _cl5WriteOperationTxn: failed
>>>>                 to write entry with csn (53be3097000000040000); db
>>>>                 error - -30994 DB_LOCK_DEADLOCK: Locker killed to
>>>>                 resolve a deadlock
>>>>                 [10/Jul/2014:14:20:01 +0800] NSMMReplicationPlugin
>>>>                 - write_changelog_and_ruv: can't add a change for
>>>>                 uid=xuehuimei,cn=users,cn=accounts,dc=abc,dc=com
>>>>                 (uniqid: 1300de84-07fa11e4-b3ddf885-593f3a7a,
>>>>                 optype: 16) to changelog csn 53be3097000000040000
>>>>                 [10/Jul/2014:14:56:51 +0800] NSMMReplicationPlugin
>>>>                 - changelog program - _cl5WriteOperationTxn: retry
>>>>                 (49) the transaction (csn=53be3939000000040000)
>>>>                 failed (rc=-30994 (DB_LOCK_DEADLOCK: Locker killed
>>>>                 to resolve a deadlock))
>>>>                 [10/Jul/2014:14:56:51 +0800] NSMMReplicationPlugin
>>>>                 - changelog program - _cl5WriteOperationTxn: failed
>>>>                 to write entry with csn (53be3939000000040000); db
>>>>                 error - -30994 DB_LOCK_DEADLOCK: Locker killed to
>>>>                 resolve a deadlock
>>>>                 [10/Jul/2014:14:56:51 +0800] NSMMReplicationPlugin
>>>>                 - write_changelog_and_ruv: can't add a change for
>>>>                 uid=websubcon04,cn=users,cn=accounts,dc=abc,dc=com
>>>>                 (uniqid: 3e39fc81-07ff11e4-b3ddf885-593f3a7a,
>>>>                 optype: 16) to changelog csn 53be3939000000040000
>>>
>>>                 This looks like
>>>                 https://fedorahosted.org/389/ticket/47409 and
>>>                 https://bugzilla.redhat.com/show_bug.cgi?id=979169
>>>
>>>                 Cause: Under certain conditions, with a mix of
>>>                 concurrent search and update and outgoing
>>>                 replication operations, there will be deadlocks in
>>>                 the changelog db, leading to error messages like this:
>>>                 NSMMReplicationPlugin - changelog program -
>>>                 _cl5WriteOperationTxn: failed to write entry with
>>>                 csn (XXXXXXX); db error - -30994 DB_LOCK_DEADLOCK:
>>>                 Locker killed to resolve a deadlock
>>>                 This is caused by a deadlock between the changelog
>>>                 readers, writers, and main database writers.
>>>
>>>                 Consequence: Update operations will fail with the
>>>                 above error message in the directory server errors log.
>>>
>>>                 Fix: A new configuration parameter is introduced:
>>>                 dn: cn=config,cn=ldbm database,cn=plugins,cn=config
>>>                 nsslapd-db-deadlock-policy: 9
>>>
>>>                 With the default policy 9 (DB_LOCK_YOUNGEST), the
>>>                 last locker gets killed when there is a deadlock. 
>>>                 In the case that this is the changelog writer, the
>>>                 write will fail, and the entire update will fail.
>>>
>>>                 Users who frequently see the above errors in the
>>>                 errors log are advised to change this setting to 6
>>>                 (DB_LOCK_MINWRITE) will which instead kill the
>>>                 locker that has the fewest write locks (that is, the
>>>                 changelog reader).  The changelog reader code has
>>>                 been changed to handle this deadlock condition and
>>>                 retry.  The setting can be changed like this:
>>>
>>>                 ldapmodify -x -D "cn=directory manager" -W <<EOF
>>>                 dn: cn=config,cn=ldbm database,cn=plugins,cn=config
>>>                 changetype: modify
>>>                 replace: nsslapd-db-deadlock-policy
>>>                 nsslapd-db-deadlock-policy: 6
>>>                 EOF
>>>
>>>                 You may ask why the default is not changed to 6. 
>>>                 The answer is that the setting will apply to _all_
>>>                 threads, so that changing this setting could cause
>>>                 regular search requests to fail, if the directory
>>>                 server is under a heavy update load.  In our
>>>                 testing, we did not see this happen, but we cannot
>>>                 guarantee that changing this value to 6 will not
>>>                 impact regular search requests.
>>>
>>>                 Result: After changing nsslapd-db-deadlock-policy to
>>>                 6, updates will succeed and no longer cause errors
>>>                 like the above.
>>>
>>>
>>>>
>>>>
>>>>                 2014-07-10 10:40 GMT+08:00 Rich Megginson
>>>>                 <rmeggins at redhat.com <mailto:rmeggins at redhat.com>>:
>>>>
>>>>                     On 07/09/2014 08:36 PM, barrykfl at gmail.com
>>>>                     <mailto:barrykfl at gmail.com> wrote:
>>>>>                     Hi :
>>>>>
>>>>>                     What is the procedure for this minor update ?
>>>>>
>>>>>                     just yum update ipa-server after stop the server?
>>>>
>>>>                     If you just want to upgrade only the LDAP
>>>>                     server, which is the component that I for sure
>>>>                     know is out of date, then yum update 389-ds-base.
>>>>
>>>>                     Or just "yum update" - in general I don't like
>>>>                     running "franken-systems" which have a mix of
>>>>                     up-to-date and out of date packages.  Note that
>>>>                     "IPA server" is composed of several packages.
>>>>
>>>>                     You do not need to stop the server.  yum/rpm
>>>>                     upgrade will restart as needed.  If you want to
>>>>                     make sure, do ipactl restart after upgrade.
>>>>
>>>>
>>>>>                     and effect of the exsitn ldap?
>>>>
>>>>                     Not sure what you mean. Upgrade should not
>>>>                     touch any config or data.
>>>>
>>>>
>>>>>
>>>>>                     As the server 2 is master of replica also , so
>>>>>                     need refo ipa-replica install ?
>>>>
>>>>                     No, you just need to perform the same upgrade
>>>>                     procedure.
>>>>
>>>>
>>>>>
>>>>>                     barry
>>>>>
>>>>>
>>>>>                     2014-07-09 22:20 GMT+08:00 Rich Megginson
>>>>>                     <rmeggins at redhat.com
>>>>>                     <mailto:rmeggins at redhat.com>>:
>>>>>
>>>>>                         On 07/08/2014 09:02 PM, barrykfl at gmail.com
>>>>>                         <mailto:barrykfl at gmail.com> wrote:
>>>>>>                         Some error i found :
>>>>>>
>>>>>>
>>>>>>                         server1.abc.com:636
>>>>>>                         <http://server1.abc.com:636>
>>>>>>                         (/etc/dirsrv/slapd-abc-COM)
>>>>>>
>>>>>>                         [29/Jun/2014:02:00:56 +0800] -
>>>>>>                         389-Directory/1.2.11.25
>>>>>>                         <http://1.2.11.25> B2013.325.1951 starting up
>>>>>>                         [29/Jun/2014:02:00:56 +0800] attrcrypt -
>>>>>>                         attrcrypt_unwrap_key: failed to unwrap
>>>>>>                         key for cipher AES
>>>>>>                         [29/Jun/2014:02:00:56 +0800] attrcrypt -
>>>>>>                         attrcrypt_cipher_init: symmetric key
>>>>>>                         failed to unwrap with the private key;
>>>>>>                         Cert might have been renewed since the
>>>>>>                         key is wrapped.  To recover the encrypted
>>>>>>                         contents, keep the wrapped symmetric key
>>>>>>                         value.
>>>>>>                         [29/Jun/2014:02:00:56 +0800] attrcrypt -
>>>>>>                         attrcrypt_unwrap_key: failed to unwrap
>>>>>>                         key for cipher 3DES
>>>>>>                         [29/Jun/2014:02:00:56 +0800] attrcrypt -
>>>>>>                         attrcrypt_cipher_init: symmetric key
>>>>>>                         failed to unwrap with the private key;
>>>>>>                         Cert might have been renewed since the
>>>>>>                         key is wrapped.  To recover the encrypted
>>>>>>                         contents, keep the wrapped symmetric key
>>>>>>                         value.
>>>>>>                         [29/Jun/2014:02:00:56 +0800] attrcrypt -
>>>>>>                         All prepared ciphers are not available.
>>>>>>                         Please disable attribute encryption.
>>>>>>                         [29/Jun/2014:02:00:56 +0800]
>>>>>>                         schema-compat-plugin - warning: no
>>>>>>                         entries set up under cn=computers,
>>>>>>                         cn=compat,dc=abc,dc=com
>>>>>>                         [29/Jun/2014:02:00:57 +0800]
>>>>>>                         schema-compat-plugin - warning: no
>>>>>>                         entries set up under cn=ng,
>>>>>>                         cn=compat,dc=abc,dc=com
>>>>>>                         [29/Jun/2014:02:00:57 +0800]
>>>>>>                         schema-compat-plugin - warning: no
>>>>>>                         entries set up under ou=sudoers,dc=abc,dc=com
>>>>>>                         [29/Jun/2014:02:00:57 +0800] - Skipping
>>>>>>                         CoS Definition cn=Password
>>>>>>                         Policy,cn=accounts,dc=abc,dc=com--no CoS
>>>>>>                         Templates found, which should be added
>>>>>>                         before the CoS Definition.
>>>>>>                         [29/Jun/2014:02:00:57 +0800]
>>>>>>                         set_krb5_creds - Could not get initial
>>>>>>                         credentials for principal
>>>>>>                         [ldap/server1.abc.com at abc.COM
>>>>>>                         <mailto:ldap/server1.abc.com at abc.COM>] in
>>>>>>                         keytab [FILE:/etc/dirsrv/ds.keytab]:
>>>>>>                         -1765328228 (Cannot contact any KDC for
>>>>>>                         requested realm)
>>>>>>                         [29/Jun/2014:02:00:58 +0800] - Skipping
>>>>>>                         CoS Definition cn=Password
>>>>>>                         Policy,cn=accounts,dc=abc,dc=com--no CoS
>>>>>>                         Templates found, which should be added
>>>>>>                         before the CoS Definition.
>>>>>>                         [29/Jun/2014:02:00:58 +0800]
>>>>>>                         slapd_ldap_sasl_interactive_bind - Error:
>>>>>>                         could not perform interactive bind for id
>>>>>>                         [] mech [GSSAPI]: LDAP error -2 (Local
>>>>>>                         error) (SASL(-1): generic failure: GSSAPI
>>>>>>                         Error: Unspecified GSS failure.  Minor
>>>>>>                         code may provide more information
>>>>>>                         (Credentials cache file '/tmp/krb5cc_492'
>>>>>>                         not found)) errno 0 (Success)
>>>>>>                         [29/Jun/2014:02:00:58 +0800]
>>>>>>                         slapi_ldap_bind - Error: could not
>>>>>>                         perform interactive bind for id [] mech
>>>>>>                         [GSSAPI]: error -2 (Local error)
>>>>>>                         [29/Jun/2014:02:00:58 +0800]
>>>>>>                         NSMMReplicationPlugin -
>>>>>>                         agmt="cn=meToserver2.abc.com
>>>>>>                         <http://meToserver2.abc.com>"
>>>>>>                         (server2:389): Replication bind with
>>>>>>                         GSSAPI auth failed: LDAP error -2 (Local
>>>>>>                         error) (SASL(-1): generic failure: GSSAPI
>>>>>>                         Error: Unspecified GSS failure.  Minor
>>>>>>                         code may provide more information
>>>>>>                         (Credentials cache file '/tmp/krb5cc_492'
>>>>>>                         not found))
>>>>>>                         [29/Jun/2014:02:00:58 +0800] - slapd
>>>>>>                         started.  Listening on All Interfaces
>>>>>>                         port 389 for LDAP requests
>>>>>>                         [29/Jun/2014:02:00:58 +0800] - Listening
>>>>>>                         on All Interfaces port 636 for LDAPS requests
>>>>>>
>>>>>>
>>>>>>                         389-Directory/1.2.11.15
>>>>>>                         <http://1.2.11.15> B2013.240.174
>>>>>>                         server2.abc.com:636
>>>>>>                         <http://server2.abc.com:636>
>>>>>>                         (/etc/dirsrv/slapd-abc-COM)
>>>>>>
>>>>>>                         [30/Jun/2014:12:51:31 +0800]
>>>>>>                         slapd_ldap_sasl_interactive_bind - Error:
>>>>>>                         could not perform interactive bind for id
>>>>>>                         [] mech [GSSAPI]: LDAP error -2 (Local
>>>>>>                         error) (SASL(-1): generic failure: GSSAPI
>>>>>>                         Error: Unspecified GSS failure.  Minor
>>>>>>                         code may provide more information (Ticket
>>>>>>                         expired)) errno 0 (Success)
>>>>>>                         [30/Jun/2014:12:51:31 +0800]
>>>>>>                         slapd_ldap_sasl_interactive_bind - Error:
>>>>>>                         could not perform interactive bind for id
>>>>>>                         [] mech [GSSAPI]: LDAP error -2 (Local
>>>>>>                         error) (SASL(-1): generic failure: GSSAPI
>>>>>>                         Error: Unspecified GSS failure.  Minor
>>>>>>                         code may provide more information (Ticket
>>>>>>                         expired)) errno 0 (Success)
>>>>>>                         [30/Jun/2014:12:51:31 +0800]
>>>>>>                         slapi_ldap_bind - Error: could not
>>>>>>                         perform interactive bind for id [] mech
>>>>>>                         [GSSAPI]: error -2 (Local error)
>>>>>>                         [30/Jun/2014:12:51:31 +0800]
>>>>>>                         NSMMReplicationPlugin -
>>>>>>                         agmt="cn=meToserver1.abc.com
>>>>>>                         <http://meToserver1.abc.com>"
>>>>>>                         (server1:389): Replication bind with
>>>>>>                         GSSAPI auth failed: LDAP error -2 (Local
>>>>>>                         error) (SASL(-1): generic failure: GSSAPI
>>>>>>                         Error: Unspecified GSS failure.  Minor
>>>>>>                         code may provide more information (Ticket
>>>>>>                         expired))
>>>>>>                         [30/Jun/2014:12:51:34 +0800]
>>>>>>                         slapd_ldap_sasl_interactive_bind - Error:
>>>>>>                         could not perform interactive bind for id
>>>>>>                         [] mech [GSSAPI]: LDAP error -2 (Local
>>>>>>                         error) (SASL(-1): generic failure: GSSAPI
>>>>>>                         Error: Unspecified GSS failure.  Minor
>>>>>>                         code may provide more information (Ticket
>>>>>>                         expired)) errno 0 (Success)
>>>>>>                         [30/Jun/2014:12:51:35 +0800]
>>>>>>                         slapd_ldap_sasl_interactive_bind - Error:
>>>>>>                         could not perform interactive bind for id
>>>>>>                         [] mech [GSSAPI]: LDAP error -2 (Local
>>>>>>                         error) (SASL(-1): generic failure: GSSAPI
>>>>>>                         Error: Unspecified GSS failure.  Minor
>>>>>>                         code may provide more information (Ticket
>>>>>>                         expired)) errno 0 (Success)
>>>>>>                         [30/Jun/2014:12:51:35 +0800]
>>>>>>                         slapi_ldap_bind - Error: could not
>>>>>>                         perform interactive bind for id [] mech
>>>>>>                         [GSSAPI]: error -2 (Local error)
>>>>>>                         [30/Jun/2014:12:51:40 +0800]
>>>>>>                         slapd_ldap_sasl_interactive_bind - Error:
>>>>>>                         could not perform interactive bind for id
>>>>>>                         [] mech [GSSAPI]: LDAP error -2 (Local
>>>>>>                         error) (SASL(-1): generic failure: GSSAPI
>>>>>>                         Error: Unspecified GSS failure.  Minor
>>>>>>                         code may provide more information (Ticket
>>>>>>                         expired)) errno 0 (Success)
>>>>>>                         [30/Jun/2014:12:51:40 +0800]
>>>>>>                         slapd_ldap_sasl_interactive_bind - Error:
>>>>>>                         could not perform interactive bind for id
>>>>>>                         [] mech [GSSAPI]: LDAP error -2 (Local
>>>>>>                         error) (SASL(-1): generic failure: GSSAPI
>>>>>>                         Error: Unspecified GSS failure.  Minor
>>>>>>                         code may provide more information (Ticket
>>>>>>                         expired)) errno 0 (Success)
>>>>>>                         [30/Jun/2014:12:51:40 +0800]
>>>>>>                         slapi_ldap_bind - Error: could not
>>>>>>                         perform interactive bind for id [] mech
>>>>>>                         [GSSAPI]: error -2 (Local error)
>>>>>>                         [30/Jun/2014:12:51:52 +0800]
>>>>>>                         NSMMReplicationPlugin -
>>>>>>                         agmt="cn=meToserver1.abc.com
>>>>>>                         <http://meToserver1.abc.com>"
>>>>>>                         (server1:389): Replication bind with
>>>>>>                         GSSAPI auth resumed
>>>>>>
>>>>>
>>>>>                         You are using an older version of 389. 
>>>>>                         The version on server2 is older than the
>>>>>                         version on server1.  Can you upgrade and
>>>>>                         see if that fixes your problems? Even if
>>>>>                         it doesn't fix your problems, it will be
>>>>>                         much easier for us to support.
>>>>>
>>>>>
>>>>>>
>>>>>>                         2014-07-09 10:55 GMT+08:00
>>>>>>                         <barrykfl at gmail.com
>>>>>>                         <mailto:barrykfl at gmail.com>>:
>>>>>>
>>>>>>                             FYI..
>>>>>>                             160: [04/Jul/2014:12:35:30 +0800]
>>>>>>                             conn=936207 fd=73 slot=73 connection
>>>>>>                             from 192.168.156.89 to 192.168.156.89
>>>>>>                             163: [04/Jul/2014:12:35:30 +0800]
>>>>>>                             conn=936207 op=-1 fd=73 closed - B1
>>>>>>
>>>>>>                             There is not abt binding but i unsure
>>>>>>                             how to fix ..
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>                             2014-07-09 2:01 GMT+08:00 Rich
>>>>>>                             Megginson <rmeggins at redhat.com
>>>>>>                             <mailto:rmeggins at redhat.com>>:
>>>>>>
>>>>>>                                 On 07/08/2014 02:16 AM,
>>>>>>                                 barrykfl at gmail.com
>>>>>>                                 <mailto:barrykfl at gmail.com> wrote:
>>>>>>>                                 Resent as size limit.
>>>>>>>
>>>>>>>
>>>>>>>                                 Here u are  server1 's access
>>>>>>>                                 log seem one side broken
>>>>>>>
>>>>>>>                                 the problem is how to make it
>>>>>>>                                 replicate again.
>>>>>>>
>>>>>>>                                 At server 1
>>>>>>>
>>>>>>>                                 it is ok  master server1 master
>>>>>>>                                 server2
>>>>>>>
>>>>>>>
>>>>>>>                                 Another side server 2 contains 2
>>>>>>>                                 ip replication.
>>>>>>>
>>>>>>>                                 ipa-replica-manage list shown
>>>>>>>                                 Can't contact LDAP server
>>>>>>>
>>>>>>>                                 I dont know why but the
>>>>>>>                                 prolematic server is sever 2 not
>>>>>>>                                 server 1
>>>>>>>
>>>>>>>                                 log of server2
>>>>>>>                                 [08/Jul/2014:16:02:40 +0800]
>>>>>>>                                 conn=3299731 fd=69 slot=69
>>>>>>>                                 connection from 192.168.15.89
>>>>>>>                                 (server1) to 192.168.15.88(server2)
>>>>>>>                                 [08/Jul/2014:16:02:40 +0800]
>>>>>>>                                 conn=3299731 op=-1 fd=69 closed - B1
>>>>>>>                                 [08/Jul/2014:16:02:40 +0800]
>>>>>>>                                 conn=3299732 fd=69 slot=69
>>>>>>>                                 connection from 192.168.15.89 to
>>>>>>>                                 192.168.15.88
>>>>>>>                                 [08/Jul/2014:16:02:40 +0800]
>>>>>>>                                 conn=3299732 op=-1 fd=69 closed - B1
>>>>>>>                                 [08/Jul/2014:16:02:41 +0800]
>>>>>>>                                 conn=3299733 fd=69 slot=69
>>>>>>>                                 connection from 192.168.15.89 to
>>>>>>>                                 192.168.15.88
>>>>>>>                                 [08/Jul/2014:16:02:41 +0800]
>>>>>>>                                 conn=3299733 op=-1 fd=69 closed - B1
>>>>>>
>>>>>>                                 You never answered my question
>>>>>>                                 below.  "Are you sure that this
>>>>>>                                 connection is a replication
>>>>>>                                 session?  Can you post all of the
>>>>>>                                 operations from the access log
>>>>>>                                 from conn=936207?"
>>>>>>
>>>>>>                                 In the future, please avoid
>>>>>>                                 spamming the list with large log
>>>>>>                                 files.  In general, it's better
>>>>>>                                 to provide excerpts from the log
>>>>>>                                 files showing the problem, paste
>>>>>>                                 them to fpaste.org
>>>>>>                                 <http://fpaste.org>, and post the
>>>>>>                                 link to the mailing list. If for
>>>>>>                                 some reason you need to post a
>>>>>>                                 large file, please use a file
>>>>>>                                 sharing service and post the link
>>>>>>                                 to the file.
>>>>>>
>>>>>>                                 Can you take a look at your
>>>>>>                                 errors log from server 1 and
>>>>>>                                 server 2 and see if there are any
>>>>>>                                 relevant errors?
>>>>>>
>>>>>>                                 If I had to guess, I would say
>>>>>>                                 that there is some sort of
>>>>>>                                 network error between server 1
>>>>>>                                 and server 2 that causes the
>>>>>>                                 excessive closed - B1. Perhaps
>>>>>>                                 there will be more information in
>>>>>>                                 the errors log.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>                                 2014-07-07 22:21 GMT+08:00 Rich
>>>>>>>                                 Megginson <rmeggins at redhat.com
>>>>>>>                                 <mailto:rmeggins at redhat.com>>:
>>>>>>>
>>>>>>>                                     On 07/04/2014 03:28 AM,
>>>>>>>                                     barrykfl at gmail.com
>>>>>>>                                     <mailto:barrykfl at gmail.com>
>>>>>>>                                     wrote:
>>>>>>>>                                     FOUND something strange
>>>>>>>>                                     that server 1 replicate to
>>>>>>>>                                     itself rather than server2
>>>>>>>>
>>>>>>>>                                     Server1 access log > Wrong
>>>>>>>>                                     [04/Jul/2014:12:35:30
>>>>>>>>                                     +0800] conn=936207 fd=73
>>>>>>>>                                     slot=73 connection from
>>>>>>>>                                     192.168.15.89( server1 )
>>>>>>>>                                      to 192.168.15.89 (server1)
>>>>>>>
>>>>>>>                                     Are you sure that this
>>>>>>>                                     connection is a replication
>>>>>>>                                     session?  Can you post all
>>>>>>>                                     of the operations from the
>>>>>>>                                     access log from conn=936207?
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                                     Server 2 access log > OK
>>>>>>>>                                     [04/Jul/2014:12:35:30
>>>>>>>>                                     +0800] conn=936208 fd=74
>>>>>>>>                                     slot=74 connection from
>>>>>>>>                                     192.168.15.89(server2) to
>>>>>>>>                                     192.168.15.88 (server2)
>>>>>>>>
>>>>>>>>
>>>>>>>>                                     2014-07-04 9:25 GMT+08:00
>>>>>>>>                                     <barrykfl at gmail.com
>>>>>>>>                                     <mailto:barrykfl at gmail.com>>:
>>>>>>>>
>>>>>>>>                                         Just sure now one side
>>>>>>>>                                         flow is broken, if u
>>>>>>>>                                         update server1 , it
>>>>>>>>                                         100% work server2 will
>>>>>>>>                                         upgrade.
>>>>>>>>                                         but if u update server2
>>>>>>>>                                         there is chance non-syn
>>>>>>>>                                         e.g it create username
>>>>>>>>                                          in server1 with posfix
>>>>>>>>                                         grp >ok
>>>>>>>>                                         but in server2 it only
>>>>>>>>                                         created posfix grp but
>>>>>>>>                                         no username /attribute
>>>>>>>>                                         it occur serveral
>>>>>>>>                                         times. I have to use
>>>>>>>>                                         command line grp del
>>>>>>>>                                         ...etc. to force del
>>>>>>>>                                         them and recreate them.,.
>>>>>>>>
>>>>>>>>                                         Result below:
>>>>>>>>
>>>>>>>>                                         server2.abc.com
>>>>>>>>                                         <http://server2.abc.com>:
>>>>>>>>                                         replica
>>>>>>>>                                           last init status: None
>>>>>>>>                                           last init ended: None
>>>>>>>>                                           last update status: 0
>>>>>>>>                                         Replica acquired
>>>>>>>>                                         successfully:
>>>>>>>>                                         Incremental update
>>>>>>>>                                         succeeded
>>>>>>>>                                           last update ended:
>>>>>>>>                                         2014-07-04 00:33:18+00:00
>>>>>>>>
>>>>>>>>                                         Directory Manager password:
>>>>>>>>
>>>>>>>>                                         server1.abc.com
>>>>>>>>                                         <http://server1.abc.com>:
>>>>>>>>                                         replica
>>>>>>>>                                           last init status: 0
>>>>>>>>                                         Total update succeeded
>>>>>>>>                                           last init ended:
>>>>>>>>                                         2014-06-20 10:07:02+00:00
>>>>>>>>                                           last update status: 0
>>>>>>>>                                         Replica acquired
>>>>>>>>                                         successfully:
>>>>>>>>                                         Incremental update
>>>>>>>>                                         succeeded
>>>>>>>>                                           last update ended:
>>>>>>>>                                         2014-07-04 01:14:19+00:00
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                                         [root@(LIVE)server2 ~]$
>>>>>>>>                                          ipactl status
>>>>>>>>                                         Directory Service: RUNNING
>>>>>>>>                                         KDC Service: RUNNING
>>>>>>>>                                         KPASSWD Service: RUNNING
>>>>>>>>                                         MEMCACHE Service: RUNNING
>>>>>>>>                                         HTTP Service: RUNNING
>>>>>>>>
>>>>>>>>
>>>>>>>>                                         2014-07-04 1:34
>>>>>>>>                                         GMT+08:00 Rob
>>>>>>>>                                         Crittenden
>>>>>>>>                                         <rcritten at redhat.com
>>>>>>>>                                         <mailto:rcritten at redhat.com>>:
>>>>>>>>
>>>>>>>>
>>>>>>>>                                             barrykfl at gmail.com
>>>>>>>>                                             <mailto:barrykfl at gmail.com>
>>>>>>>>                                             wrote:
>>>>>>>>                                             > Yes they are
>>>>>>>>                                             running. Server 1
>>>>>>>>                                             can syn to server2
>>>>>>>>                                             but error at server 2
>>>>>>>>                                             > like this.
>>>>>>>>
>>>>>>>>                                             How do you know
>>>>>>>>                                             server 1 is syncing
>>>>>>>>                                             with server 2?
>>>>>>>>
>>>>>>>>                                             On server 1 I'd run:
>>>>>>>>
>>>>>>>>                                             ipa-replica-manage
>>>>>>>>                                             list -v `hostname`
>>>>>>>>
>>>>>>>>                                             This will show the
>>>>>>>>                                             replication status.
>>>>>>>>
>>>>>>>>                                             And what does
>>>>>>>>                                             ipactl status show
>>>>>>>>                                             on server 2?
>>>>>>>>
>>>>>>>>                                             rob
>>>>>>>>
>>>>>>>>                                             >
>>>>>>>>                                             > 2014/7/3 下午
>>>>>>>>                                             10:14 於 "Rob
>>>>>>>>                                             Crittenden"
>>>>>>>>                                             <rcritten at redhat.com <mailto:rcritten at redhat.com>
>>>>>>>>                                             >
>>>>>>>>                                             <mailto:rcritten at redhat.com
>>>>>>>>                                             <mailto:rcritten at redhat.com>>>
>>>>>>>>                                             寫道:
>>>>>>>>                                             >
>>>>>>>>                                             > Please keep
>>>>>>>>                                             relies on the list.
>>>>>>>>                                             >
>>>>>>>>                                             >
>>>>>>>>                                             barrykfl at gmail.com
>>>>>>>>                                             <mailto:barrykfl at gmail.com>
>>>>>>>>                                             <mailto:barrykfl at gmail.com
>>>>>>>>                                             <mailto:barrykfl at gmail.com>>
>>>>>>>>                                             wrote:
>>>>>>>>                                             >     > I saw the
>>>>>>>>                                             error beloe and
>>>>>>>>                                             errpr log is it
>>>>>>>>                                             related ?
>>>>>>>>                                             >     >
>>>>>>>>                                             >     >
>>>>>>>>                                             29/Jun/2014:02:00:58 +0800]
>>>>>>>>                                             slapd_ldap_sasl_interactive_bind
>>>>>>>>                                             - Error:
>>>>>>>>                                             >     > could not
>>>>>>>>                                             perform interactive
>>>>>>>>                                             bind for id [] mech
>>>>>>>>                                             [GSSAPI]: LDAP error
>>>>>>>>                                             >     > -2 (Local
>>>>>>>>                                             error) (SASL(-1):
>>>>>>>>                                             generic failure:
>>>>>>>>                                             GSSAPI Error:
>>>>>>>>                                             Unspecified
>>>>>>>>                                             >     > GSS
>>>>>>>>                                             failure.  Minor
>>>>>>>>                                             code may provide
>>>>>>>>                                             more information
>>>>>>>>                                             (Credentials
>>>>>>>>                                             >     cache
>>>>>>>>                                             >     > file
>>>>>>>>                                             '/tmp/krb5cc_492'
>>>>>>>>                                             not found)) errno 0
>>>>>>>>                                             (Success)
>>>>>>>>                                             >     >
>>>>>>>>                                             [29/Jun/2014:02:00:58
>>>>>>>>                                             +0800]
>>>>>>>>                                             slapi_ldap_bind -
>>>>>>>>                                             Error: could not
>>>>>>>>                                             > perform
>>>>>>>>                                             >     > interactive
>>>>>>>>                                             bind for id [] mech
>>>>>>>>                                             [GSSAPI]: error -2
>>>>>>>>                                             (Local error)
>>>>>>>>                                             >
>>>>>>>>                                             >     I believe
>>>>>>>>                                             this is fairly
>>>>>>>>                                             normal on a new
>>>>>>>>                                             startup. It has to
>>>>>>>>                                             start
>>>>>>>>                                             > somewhere. The
>>>>>>>>                                             expired ticket
>>>>>>>>                                             errors below are
>>>>>>>>                                             unexpected since there
>>>>>>>>                                             >     are so many
>>>>>>>>                                             of them. Is your
>>>>>>>>                                             KDC running?
>>>>>>>>                                             >
>>>>>>>>                                             > ipactl status
>>>>>>>>                                             >
>>>>>>>>                                             >     rob
>>>>>>>>                                             >
>>>>>>>>                                             >     >
>>>>>>>>                                             >     >
>>>>>>>>                                             >     > 2014-07-02
>>>>>>>>                                             14:15 GMT+08:00
>>>>>>>>                                             <barrykfl at gmail.com
>>>>>>>>                                             <mailto:barrykfl at gmail.com>
>>>>>>>>                                             >
>>>>>>>>                                             <mailto:barrykfl at gmail.com
>>>>>>>>                                             <mailto:barrykfl at gmail.com>>
>>>>>>>>                                             <mailto:barrykfl at gmail.com
>>>>>>>>                                             <mailto:barrykfl at gmail.com>
>>>>>>>>                                             >
>>>>>>>>                                             <mailto:barrykfl at gmail.com
>>>>>>>>                                             <mailto:barrykfl at gmail.com>>>>:
>>>>>>>>                                             >     >
>>>>>>>>                                             >     >
>>>>>>>>                                             >     >     this is
>>>>>>>>                                             the error log i
>>>>>>>>                                             found at 2.abc.com
>>>>>>>>                                             <http://2.abc.com>
>>>>>>>>                                             <http://2.abc.com>
>>>>>>>>                                             >    
>>>>>>>>                                             <http://2.abc.com>
>>>>>>>>                                             >     >
>>>>>>>>                                             >     >
>>>>>>>>                                             [30/Jun/2014:12:51:31
>>>>>>>>                                             +0800]
>>>>>>>>                                             slapd_ldap_sasl_interactive_bind
>>>>>>>>                                             -
>>>>>>>>                                             >     >     Error:
>>>>>>>>                                             could not perform
>>>>>>>>                                             interactive bind
>>>>>>>>                                             for id [] mech
>>>>>>>>                                             [GSSAPI]:
>>>>>>>>                                             >     >     LDAP
>>>>>>>>                                             error -2 (Local
>>>>>>>>                                             error) (SASL(-1):
>>>>>>>>                                             generic failure: GSSAPI
>>>>>>>>                                             >     >     Error:
>>>>>>>>                                             Unspecified GSS
>>>>>>>>                                             failure.  Minor
>>>>>>>>                                             code may provide more
>>>>>>>>                                             >     > information
>>>>>>>>                                             (Ticket expired))
>>>>>>>>                                             errno 0 (Success)
>>>>>>>>                                             >     >
>>>>>>>>                                             [30/Jun/2014:12:51:31
>>>>>>>>                                             +0800]
>>>>>>>>                                             slapd_ldap_sasl_interactive_bind
>>>>>>>>                                             -
>>>>>>>>                                             >     >     Error:
>>>>>>>>                                             could not perform
>>>>>>>>                                             interactive bind
>>>>>>>>                                             for id [] mech
>>>>>>>>                                             [GSSAPI]:
>>>>>>>>                                             >     >     LDAP
>>>>>>>>                                             error -2 (Local
>>>>>>>>                                             error) (SASL(-1):
>>>>>>>>                                             generic failure: GSSAPI
>>>>>>>>                                             >     >     Error:
>>>>>>>>                                             Unspecified GSS
>>>>>>>>                                             failure.  Minor
>>>>>>>>                                             code may provide more
>>>>>>>>                                             >     > information
>>>>>>>>                                             (Ticket expired))
>>>>>>>>                                             errno 0 (Success)
>>>>>>>>                                             >     >
>>>>>>>>                                             [30/Jun/2014:12:51:31
>>>>>>>>                                             +0800]
>>>>>>>>                                             slapi_ldap_bind -
>>>>>>>>                                             Error: could not
>>>>>>>>                                             >     >     perform
>>>>>>>>                                             interactive bind
>>>>>>>>                                             for id [] mech
>>>>>>>>                                             [GSSAPI]: error -2
>>>>>>>>                                             > (Local error)
>>>>>>>>                                             >     >
>>>>>>>>                                             [30/Jun/2014:12:51:31
>>>>>>>>                                             +0800]
>>>>>>>>                                             NSMMReplicationPlugin -
>>>>>>>>                                             >     >    
>>>>>>>>                                             agmt="cn=meTo1.abc.com
>>>>>>>>                                             <http://meTo1.abc.com>
>>>>>>>>                                             <http://meTo1.abc.com>
>>>>>>>>                                             >    
>>>>>>>>                                             <http://meTo1.abc.com>"
>>>>>>>>                                             (central:389):
>>>>>>>>                                             >     > Replication
>>>>>>>>                                             bind with GSSAPI
>>>>>>>>                                             auth failed: LDAP
>>>>>>>>                                             error -2 (Local
>>>>>>>>                                             >     >     error)
>>>>>>>>                                             (SASL(-1): generic
>>>>>>>>                                             failure: GSSAPI
>>>>>>>>                                             Error: Unspecified GSS
>>>>>>>>                                             >     >    
>>>>>>>>                                             failure.  Minor
>>>>>>>>                                             code may provide
>>>>>>>>                                             more information
>>>>>>>>                                             (Ticket
>>>>>>>>                                             > expired))
>>>>>>>>                                             >     >
>>>>>>>>                                             [30/Jun/2014:12:51:34
>>>>>>>>                                             +0800]
>>>>>>>>                                             slapd_ldap_sasl_interactive_bind
>>>>>>>>                                             -
>>>>>>>>                                             >     >     Error:
>>>>>>>>                                             could not perform
>>>>>>>>                                             interactive bind
>>>>>>>>                                             for id [] mech
>>>>>>>>                                             [GSSAPI]:
>>>>>>>>                                             >     >     LDAP
>>>>>>>>                                             error -2 (Local
>>>>>>>>                                             error) (SASL(-1):
>>>>>>>>                                             generic failure: GSSAPI
>>>>>>>>                                             >     >     Error:
>>>>>>>>                                             Unspecified GSS
>>>>>>>>                                             failure.  Minor
>>>>>>>>                                             code may provide more
>>>>>>>>                                             >     > information
>>>>>>>>                                             (Ticket expired))
>>>>>>>>                                             errno 0 (Success)
>>>>>>>>                                             >     >
>>>>>>>>                                             [30/Jun/2014:12:51:35
>>>>>>>>                                             +0800]
>>>>>>>>                                             slapd_ldap_sasl_interactive_bind
>>>>>>>>                                             -
>>>>>>>>                                             >     >     Error:
>>>>>>>>                                             could not perform
>>>>>>>>                                             interactive bind
>>>>>>>>                                             for id [] mech
>>>>>>>>                                             [GSSAPI]:
>>>>>>>>                                             >     >     LDAP
>>>>>>>>                                             error -2 (Local
>>>>>>>>                                             error) (SASL(-1):
>>>>>>>>                                             generic failure: GSSAPI
>>>>>>>>                                             >     >     Error:
>>>>>>>>                                             Unspecified GSS
>>>>>>>>                                             failure.  Minor
>>>>>>>>                                             code may provide more
>>>>>>>>                                             >     > information
>>>>>>>>                                             (Ticket expired))
>>>>>>>>                                             errno 0 (Success)
>>>>>>>>                                             >     >
>>>>>>>>                                             [30/Jun/2014:12:51:35
>>>>>>>>                                             +0800]
>>>>>>>>                                             slapi_ldap_bind -
>>>>>>>>                                             Error: could not
>>>>>>>>                                             >     >     perform
>>>>>>>>                                             interactive bind
>>>>>>>>                                             for id [] mech
>>>>>>>>                                             [GSSAPI]: error -2
>>>>>>>>                                             > (Local error)
>>>>>>>>                                             >     >
>>>>>>>>                                             [30/Jun/2014:12:51:40
>>>>>>>>                                             +0800]
>>>>>>>>                                             slapd_ldap_sasl_interactive_bind
>>>>>>>>                                             -
>>>>>>>>                                             >     >     Error:
>>>>>>>>                                             could not perform
>>>>>>>>                                             interactive bind
>>>>>>>>                                             for id [] mech
>>>>>>>>                                             [GSSAPI]:
>>>>>>>>                                             >     >     LDAP
>>>>>>>>                                             error -2 (Local
>>>>>>>>                                             error) (SASL(-1):
>>>>>>>>                                             generic failure: GSSAPI
>>>>>>>>                                             >     >     Error:
>>>>>>>>                                             Unspecified GSS
>>>>>>>>                                             failure.  Minor
>>>>>>>>                                             code may provide more
>>>>>>>>                                             >     > information
>>>>>>>>                                             (Ticket expired))
>>>>>>>>                                             errno 0 (Success)
>>>>>>>>                                             >     >
>>>>>>>>                                             [30/Jun/2014:12:51:40
>>>>>>>>                                             +0800]
>>>>>>>>                                             slapd_ldap_sasl_interactive_bind
>>>>>>>>                                             -
>>>>>>>>                                             >     >     Error:
>>>>>>>>                                             could not perform
>>>>>>>>                                             interactive bind
>>>>>>>>                                             for id [] mech
>>>>>>>>                                             [GSSAPI]:
>>>>>>>>                                             >     >     LDAP
>>>>>>>>                                             error -2 (Local
>>>>>>>>                                             error) (SASL(-1):
>>>>>>>>                                             generic failure: GSSAPI
>>>>>>>>                                             >     >     Error:
>>>>>>>>                                             Unspecified GSS
>>>>>>>>                                             failure.  Minor
>>>>>>>>                                             code may provide more
>>>>>>>>                                             >     > information
>>>>>>>>                                             (Ticket expired))
>>>>>>>>                                             errno 0 (Success)
>>>>>>>>                                             >     >
>>>>>>>>                                             [30/Jun/2014:12:51:40
>>>>>>>>                                             +0800]
>>>>>>>>                                             slapi_ldap_bind -
>>>>>>>>                                             Error: could not
>>>>>>>>                                             >     >     perform
>>>>>>>>                                             interactive bind
>>>>>>>>                                             for id [] mech
>>>>>>>>                                             [GSSAPI]: error -2
>>>>>>>>                                             > (Local error)
>>>>>>>>                                             >     >
>>>>>>>>                                             >     >
>>>>>>>>                                             >     >    
>>>>>>>>                                             2014-07-02 12:32
>>>>>>>>                                             GMT+08:00
>>>>>>>>                                             <barrykfl at gmail.com
>>>>>>>>                                             <mailto:barrykfl at gmail.com>
>>>>>>>>                                             >
>>>>>>>>                                             <mailto:barrykfl at gmail.com
>>>>>>>>                                             <mailto:barrykfl at gmail.com>>
>>>>>>>>                                             >     >
>>>>>>>>                                             <mailto:barrykfl at gmail.com
>>>>>>>>                                             <mailto:barrykfl at gmail.com>
>>>>>>>>                                             <mailto:barrykfl at gmail.com
>>>>>>>>                                             <mailto:barrykfl at gmail.com>>>>:
>>>>>>>>                                             > >
>>>>>>>>                                             >     >         yes
>>>>>>>>                                             on node 1 it is
>>>>>>>>                                             happening only
>>>>>>>>                                             node2 fail connect
>>>>>>>>                                             >     >
>>>>>>>>                                             >     >
>>>>>>>>                                             ipa-replica-manage
>>>>>>>>                                             list 2.abc.com
>>>>>>>>                                             <http://2.abc.com>
>>>>>>>>                                             <http://2.abc.com>
>>>>>>>>                                             >    
>>>>>>>>                                             <http://2.abc.com>
>>>>>>>>                                             >     > Directory
>>>>>>>>                                             Manager password:
>>>>>>>>                                             >     >
>>>>>>>>                                             >     > 1.abc.com
>>>>>>>>                                             <http://1.abc.com>
>>>>>>>>                                             <http://1.abc.com>
>>>>>>>>                                             <http://1.abc.com>:
>>>>>>>>                                             replica
>>>>>>>>                                             > >
>>>>>>>>                                             >     >
>>>>>>>>                                             >     >
>>>>>>>>                                             >     > 2014-06-30
>>>>>>>>                                             20:59 GMT+08:00 Rob
>>>>>>>>                                             Crittenden
>>>>>>>>                                             >    
>>>>>>>>                                             <rcritten at redhat.com <mailto:rcritten at redhat.com>
>>>>>>>>                                             <mailto:rcritten at redhat.com
>>>>>>>>                                             <mailto:rcritten at redhat.com>>
>>>>>>>>                                             >     >
>>>>>>>>                                             <mailto:rcritten at redhat.com
>>>>>>>>                                             <mailto:rcritten at redhat.com>
>>>>>>>>                                             <mailto:rcritten at redhat.com
>>>>>>>>                                             <mailto:rcritten at redhat.com>>>>:
>>>>>>>>                                             > >
>>>>>>>>                                             >     > Barry wrote:
>>>>>>>>                                             >     > > Hi:
>>>>>>>>                                             >     >
>>>>>>>>
>>>                 ...
>>>
>>
>>
>>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/freeipa-users/attachments/20140714/f9a9da85/attachment.htm>


More information about the Freeipa-users mailing list