[Freeipa-users] ipa-replica-manage list fail on server 2

Rich Megginson rmeggins at redhat.com
Fri Jul 11 13:54:59 UTC 2014


On 07/11/2014 02:01 AM, barrykfl at gmail.com wrote:
> should i change all server nsslapd-db-deadlock-policy: 6 in config 
>  for problematic server only or all servers need restart?

Only for the servers that are getting the "DB_LOCK_DEADLOCK: Locker 
killed to resolve a deadlock" message in the errors log.

But this does not explain the "cant contact ldap server" errors.

Which ipa commands give the "cant contact ldap server" errors?

>
>
> 2014-07-11 15:53 GMT+08:00 <barrykfl at gmail.com 
> <mailto:barrykfl at gmail.com>>:
>
>     At server 2 there is a error:
>
>
>     [10/Jul/2014:12:29:59 +0800] NSMMReplicationPlugin -
>     agmt="cn=meToserver1.abc.com <http://meToserver1.abc.com>"
>     (central:389): Replication bind with GSSAPI auth failed: LDAP
>     error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error:
>     Unspecified GSS failure.  Minor code may provide more information
>     (Credentials cache file '/tmp/krb5cc_494' not found))
>
>
>     2014-07-11 10:26 GMT+08:00 <barrykfl at gmail.com
>     <mailto:barrykfl at gmail.com>>:
>
>         Yes ,
>         still get "cant contact ldap server" after upgrading both servers.
>
>
>         2014-07-10 23:18 GMT+08:00 Rich Megginson <rmeggins at redhat.com
>         <mailto:rmeggins at redhat.com>>:
>
>             On 07/10/2014 09:15 AM, barrykfl at gmail.com
>             <mailto:barrykfl at gmail.com> wrote:
>>
>>             But any hint that server 2 say cant contact ldap server
>>             if type ipa command?
>>
>
>             Please keep replies on list.
>
>             You still get "cant contact ldap server" after upgrading
>             both servers?
>
>>             2014/7/10 下午10:25 於 "Rich Megginson"
>>             <rmeggins at redhat.com <mailto:rmeggins at redhat.com>> 寫道:
>>
>>                 On 07/10/2014 01:14 AM, barrykfl at gmail.com
>>                 <mailto:barrykfl at gmail.com> wrote:
>>>                 Tried and now two version same ....but seem same
>>>                 situation.
>>>
>>>                 i found a related error log that server1 has account
>>>                 after added user but not replicated to server2. Is
>>>                 it too fast on UI clicking ? as i exp once that
>>>                 click very
>>>                 fast twice add and edit user may cause server 2 no
>>>                 record.
>>>
>>>
>>>                 [10/Jul/2014:14:20:01 +0800] NSMMReplicationPlugin -
>>>                 changelog program - _cl5WriteOperationTxn: retry
>>>                 (49) the transaction (csn=53be3097000000040000)
>>>                 failed (rc=-30994 (DB_LOCK_DEADLOCK: Locker killed
>>>                 to resolve a deadlock))
>>>                 [10/Jul/2014:14:20:01 +0800] NSMMReplicationPlugin -
>>>                 changelog program - _cl5WriteOperationTxn: failed to
>>>                 write entry with csn (53be3097000000040000); db
>>>                 error - -30994 DB_LOCK_DEADLOCK: Locker killed to
>>>                 resolve a deadlock
>>>                 [10/Jul/2014:14:20:01 +0800] NSMMReplicationPlugin -
>>>                 write_changelog_and_ruv: can't add a change for
>>>                 uid=xuehuimei,cn=users,cn=accounts,dc=abc,dc=com
>>>                 (uniqid: 1300de84-07fa11e4-b3ddf885-593f3a7a,
>>>                 optype: 16) to changelog csn 53be3097000000040000
>>>                 [10/Jul/2014:14:56:51 +0800] NSMMReplicationPlugin -
>>>                 changelog program - _cl5WriteOperationTxn: retry
>>>                 (49) the transaction (csn=53be3939000000040000)
>>>                 failed (rc=-30994 (DB_LOCK_DEADLOCK: Locker killed
>>>                 to resolve a deadlock))
>>>                 [10/Jul/2014:14:56:51 +0800] NSMMReplicationPlugin -
>>>                 changelog program - _cl5WriteOperationTxn: failed to
>>>                 write entry with csn (53be3939000000040000); db
>>>                 error - -30994 DB_LOCK_DEADLOCK: Locker killed to
>>>                 resolve a deadlock
>>>                 [10/Jul/2014:14:56:51 +0800] NSMMReplicationPlugin -
>>>                 write_changelog_and_ruv: can't add a change for
>>>                 uid=websubcon04,cn=users,cn=accounts,dc=abc,dc=com
>>>                 (uniqid: 3e39fc81-07ff11e4-b3ddf885-593f3a7a,
>>>                 optype: 16) to changelog csn 53be3939000000040000
>>
>>                 This looks like
>>                 https://fedorahosted.org/389/ticket/47409 and
>>                 https://bugzilla.redhat.com/show_bug.cgi?id=979169
>>
>>                 Cause: Under certain conditions, with a mix of
>>                 concurrent search and update and outgoing replication
>>                 operations, there will be deadlocks in the changelog
>>                 db, leading to error messages like this:
>>                 NSMMReplicationPlugin - changelog program -
>>                 _cl5WriteOperationTxn: failed to write entry with csn
>>                 (XXXXXXX); db error - -30994 DB_LOCK_DEADLOCK: Locker
>>                 killed to resolve a deadlock
>>                 This is caused by a deadlock between the changelog
>>                 readers, writers, and main database writers.
>>
>>                 Consequence: Update operations will fail with the
>>                 above error message in the directory server errors log.
>>
>>                 Fix: A new configuration parameter is introduced:
>>                 dn: cn=config,cn=ldbm database,cn=plugins,cn=config
>>                 nsslapd-db-deadlock-policy: 9
>>
>>                 With the default policy 9 (DB_LOCK_YOUNGEST), the
>>                 last locker gets killed when there is a deadlock. In
>>                 the case that this is the changelog writer, the write
>>                 will fail, and the entire update will fail.
>>
>>                 Users who frequently see the above errors in the
>>                 errors log are advised to change this setting to 6
>>                 (DB_LOCK_MINWRITE) will which instead kill the locker
>>                 that has the fewest write locks (that is, the
>>                 changelog reader).  The changelog reader code has
>>                 been changed to handle this deadlock condition and
>>                 retry.  The setting can be changed like this:
>>
>>                 ldapmodify -x -D "cn=directory manager" -W <<EOF
>>                 dn: cn=config,cn=ldbm database,cn=plugins,cn=config
>>                 changetype: modify
>>                 replace: nsslapd-db-deadlock-policy
>>                 nsslapd-db-deadlock-policy: 6
>>                 EOF
>>
>>                 You may ask why the default is not changed to 6.  The
>>                 answer is that the setting will apply to _all_
>>                 threads, so that changing this setting could cause
>>                 regular search requests to fail, if the directory
>>                 server is under a heavy update load.  In our testing,
>>                 we did not see this happen, but we cannot guarantee
>>                 that changing this value to 6 will not impact regular
>>                 search requests.
>>
>>                 Result: After changing nsslapd-db-deadlock-policy to
>>                 6, updates will succeed and no longer cause errors
>>                 like the above.
>>
>>
>>>
>>>
>>>                 2014-07-10 10:40 GMT+08:00 Rich Megginson
>>>                 <rmeggins at redhat.com <mailto:rmeggins at redhat.com>>:
>>>
>>>                     On 07/09/2014 08:36 PM, barrykfl at gmail.com
>>>                     <mailto:barrykfl at gmail.com> wrote:
>>>>                     Hi :
>>>>
>>>>                     What is the procedure for this minor update ?
>>>>
>>>>                     just yum update ipa-server after stop the server?
>>>
>>>                     If you just want to upgrade only the LDAP
>>>                     server, which is the component that I for sure
>>>                     know is out of date, then yum update 389-ds-base.
>>>
>>>                     Or just "yum update" - in general I don't like
>>>                     running "franken-systems" which have a mix of
>>>                     up-to-date and out of date packages.  Note that
>>>                     "IPA server" is composed of several packages.
>>>
>>>                     You do not need to stop the server.  yum/rpm
>>>                     upgrade will restart as needed.  If you want to
>>>                     make sure, do ipactl restart after upgrade.
>>>
>>>
>>>>                     and effect of the exsitn ldap?
>>>
>>>                     Not sure what you mean. Upgrade should not touch
>>>                     any config or data.
>>>
>>>
>>>>
>>>>                     As the server 2 is master of replica also , so
>>>>                     need refo ipa-replica install ?
>>>
>>>                     No, you just need to perform the same upgrade
>>>                     procedure.
>>>
>>>
>>>>
>>>>                     barry
>>>>
>>>>
>>>>                     2014-07-09 22:20 GMT+08:00 Rich Megginson
>>>>                     <rmeggins at redhat.com <mailto:rmeggins at redhat.com>>:
>>>>
>>>>                         On 07/08/2014 09:02 PM, barrykfl at gmail.com
>>>>                         <mailto:barrykfl at gmail.com> wrote:
>>>>>                         Some error i found :
>>>>>
>>>>>
>>>>>                         server1.abc.com:636
>>>>>                         <http://server1.abc.com:636>
>>>>>                         (/etc/dirsrv/slapd-abc-COM)
>>>>>
>>>>>                         [29/Jun/2014:02:00:56 +0800] -
>>>>>                         389-Directory/1.2.11.25 <http://1.2.11.25>
>>>>>                         B2013.325.1951 starting up
>>>>>                         [29/Jun/2014:02:00:56 +0800] attrcrypt -
>>>>>                         attrcrypt_unwrap_key: failed to unwrap key
>>>>>                         for cipher AES
>>>>>                         [29/Jun/2014:02:00:56 +0800] attrcrypt -
>>>>>                         attrcrypt_cipher_init: symmetric key
>>>>>                         failed to unwrap with the private key;
>>>>>                         Cert might have been renewed since the key
>>>>>                         is wrapped.  To recover the encrypted
>>>>>                         contents, keep the wrapped symmetric key
>>>>>                         value.
>>>>>                         [29/Jun/2014:02:00:56 +0800] attrcrypt -
>>>>>                         attrcrypt_unwrap_key: failed to unwrap key
>>>>>                         for cipher 3DES
>>>>>                         [29/Jun/2014:02:00:56 +0800] attrcrypt -
>>>>>                         attrcrypt_cipher_init: symmetric key
>>>>>                         failed to unwrap with the private key;
>>>>>                         Cert might have been renewed since the key
>>>>>                         is wrapped.  To recover the encrypted
>>>>>                         contents, keep the wrapped symmetric key
>>>>>                         value.
>>>>>                         [29/Jun/2014:02:00:56 +0800] attrcrypt -
>>>>>                         All prepared ciphers are not available.
>>>>>                         Please disable attribute encryption.
>>>>>                         [29/Jun/2014:02:00:56 +0800]
>>>>>                         schema-compat-plugin - warning: no entries
>>>>>                         set up under cn=computers,
>>>>>                         cn=compat,dc=abc,dc=com
>>>>>                         [29/Jun/2014:02:00:57 +0800]
>>>>>                         schema-compat-plugin - warning: no entries
>>>>>                         set up under cn=ng, cn=compat,dc=abc,dc=com
>>>>>                         [29/Jun/2014:02:00:57 +0800]
>>>>>                         schema-compat-plugin - warning: no entries
>>>>>                         set up under ou=sudoers,dc=abc,dc=com
>>>>>                         [29/Jun/2014:02:00:57 +0800] - Skipping
>>>>>                         CoS Definition cn=Password
>>>>>                         Policy,cn=accounts,dc=abc,dc=com--no CoS
>>>>>                         Templates found, which should be added
>>>>>                         before the CoS Definition.
>>>>>                         [29/Jun/2014:02:00:57 +0800]
>>>>>                         set_krb5_creds - Could not get initial
>>>>>                         credentials for principal
>>>>>                         [ldap/server1.abc.com at abc.COM
>>>>>                         <mailto:ldap/server1.abc.com at abc.COM>] in
>>>>>                         keytab [FILE:/etc/dirsrv/ds.keytab]:
>>>>>                         -1765328228 (Cannot contact any KDC for
>>>>>                         requested realm)
>>>>>                         [29/Jun/2014:02:00:58 +0800] - Skipping
>>>>>                         CoS Definition cn=Password
>>>>>                         Policy,cn=accounts,dc=abc,dc=com--no CoS
>>>>>                         Templates found, which should be added
>>>>>                         before the CoS Definition.
>>>>>                         [29/Jun/2014:02:00:58 +0800]
>>>>>                         slapd_ldap_sasl_interactive_bind - Error:
>>>>>                         could not perform interactive bind for id
>>>>>                         [] mech [GSSAPI]: LDAP error -2 (Local
>>>>>                         error) (SASL(-1): generic failure: GSSAPI
>>>>>                         Error: Unspecified GSS failure.  Minor
>>>>>                         code may provide more information
>>>>>                         (Credentials cache file '/tmp/krb5cc_492'
>>>>>                         not found)) errno 0 (Success)
>>>>>                         [29/Jun/2014:02:00:58 +0800]
>>>>>                         slapi_ldap_bind - Error: could not perform
>>>>>                         interactive bind for id [] mech [GSSAPI]:
>>>>>                         error -2 (Local error)
>>>>>                         [29/Jun/2014:02:00:58 +0800]
>>>>>                         NSMMReplicationPlugin -
>>>>>                         agmt="cn=meToserver2.abc.com
>>>>>                         <http://meToserver2.abc.com>"
>>>>>                         (server2:389): Replication bind with
>>>>>                         GSSAPI auth failed: LDAP error -2 (Local
>>>>>                         error) (SASL(-1): generic failure: GSSAPI
>>>>>                         Error: Unspecified GSS failure.  Minor
>>>>>                         code may provide more information
>>>>>                         (Credentials cache file '/tmp/krb5cc_492'
>>>>>                         not found))
>>>>>                         [29/Jun/2014:02:00:58 +0800] - slapd
>>>>>                         started.  Listening on All Interfaces port
>>>>>                         389 for LDAP requests
>>>>>                         [29/Jun/2014:02:00:58 +0800] - Listening
>>>>>                         on All Interfaces port 636 for LDAPS requests
>>>>>
>>>>>
>>>>>                         389-Directory/1.2.11.15 <http://1.2.11.15>
>>>>>                         B2013.240.174
>>>>>                         server2.abc.com:636
>>>>>                         <http://server2.abc.com:636>
>>>>>                         (/etc/dirsrv/slapd-abc-COM)
>>>>>
>>>>>                         [30/Jun/2014:12:51:31 +0800]
>>>>>                         slapd_ldap_sasl_interactive_bind - Error:
>>>>>                         could not perform interactive bind for id
>>>>>                         [] mech [GSSAPI]: LDAP error -2 (Local
>>>>>                         error) (SASL(-1): generic failure: GSSAPI
>>>>>                         Error: Unspecified GSS failure.  Minor
>>>>>                         code may provide more information (Ticket
>>>>>                         expired)) errno 0 (Success)
>>>>>                         [30/Jun/2014:12:51:31 +0800]
>>>>>                         slapd_ldap_sasl_interactive_bind - Error:
>>>>>                         could not perform interactive bind for id
>>>>>                         [] mech [GSSAPI]: LDAP error -2 (Local
>>>>>                         error) (SASL(-1): generic failure: GSSAPI
>>>>>                         Error: Unspecified GSS failure.  Minor
>>>>>                         code may provide more information (Ticket
>>>>>                         expired)) errno 0 (Success)
>>>>>                         [30/Jun/2014:12:51:31 +0800]
>>>>>                         slapi_ldap_bind - Error: could not perform
>>>>>                         interactive bind for id [] mech [GSSAPI]:
>>>>>                         error -2 (Local error)
>>>>>                         [30/Jun/2014:12:51:31 +0800]
>>>>>                         NSMMReplicationPlugin -
>>>>>                         agmt="cn=meToserver1.abc.com
>>>>>                         <http://meToserver1.abc.com>"
>>>>>                         (server1:389): Replication bind with
>>>>>                         GSSAPI auth failed: LDAP error -2 (Local
>>>>>                         error) (SASL(-1): generic failure: GSSAPI
>>>>>                         Error: Unspecified GSS failure.  Minor
>>>>>                         code may provide more information (Ticket
>>>>>                         expired))
>>>>>                         [30/Jun/2014:12:51:34 +0800]
>>>>>                         slapd_ldap_sasl_interactive_bind - Error:
>>>>>                         could not perform interactive bind for id
>>>>>                         [] mech [GSSAPI]: LDAP error -2 (Local
>>>>>                         error) (SASL(-1): generic failure: GSSAPI
>>>>>                         Error: Unspecified GSS failure.  Minor
>>>>>                         code may provide more information (Ticket
>>>>>                         expired)) errno 0 (Success)
>>>>>                         [30/Jun/2014:12:51:35 +0800]
>>>>>                         slapd_ldap_sasl_interactive_bind - Error:
>>>>>                         could not perform interactive bind for id
>>>>>                         [] mech [GSSAPI]: LDAP error -2 (Local
>>>>>                         error) (SASL(-1): generic failure: GSSAPI
>>>>>                         Error: Unspecified GSS failure.  Minor
>>>>>                         code may provide more information (Ticket
>>>>>                         expired)) errno 0 (Success)
>>>>>                         [30/Jun/2014:12:51:35 +0800]
>>>>>                         slapi_ldap_bind - Error: could not perform
>>>>>                         interactive bind for id [] mech [GSSAPI]:
>>>>>                         error -2 (Local error)
>>>>>                         [30/Jun/2014:12:51:40 +0800]
>>>>>                         slapd_ldap_sasl_interactive_bind - Error:
>>>>>                         could not perform interactive bind for id
>>>>>                         [] mech [GSSAPI]: LDAP error -2 (Local
>>>>>                         error) (SASL(-1): generic failure: GSSAPI
>>>>>                         Error: Unspecified GSS failure.  Minor
>>>>>                         code may provide more information (Ticket
>>>>>                         expired)) errno 0 (Success)
>>>>>                         [30/Jun/2014:12:51:40 +0800]
>>>>>                         slapd_ldap_sasl_interactive_bind - Error:
>>>>>                         could not perform interactive bind for id
>>>>>                         [] mech [GSSAPI]: LDAP error -2 (Local
>>>>>                         error) (SASL(-1): generic failure: GSSAPI
>>>>>                         Error: Unspecified GSS failure.  Minor
>>>>>                         code may provide more information (Ticket
>>>>>                         expired)) errno 0 (Success)
>>>>>                         [30/Jun/2014:12:51:40 +0800]
>>>>>                         slapi_ldap_bind - Error: could not perform
>>>>>                         interactive bind for id [] mech [GSSAPI]:
>>>>>                         error -2 (Local error)
>>>>>                         [30/Jun/2014:12:51:52 +0800]
>>>>>                         NSMMReplicationPlugin -
>>>>>                         agmt="cn=meToserver1.abc.com
>>>>>                         <http://meToserver1.abc.com>"
>>>>>                         (server1:389): Replication bind with
>>>>>                         GSSAPI auth resumed
>>>>>
>>>>
>>>>                         You are using an older version of 389.  The
>>>>                         version on server2 is older than the
>>>>                         version on server1.  Can you upgrade and
>>>>                         see if that fixes your problems? Even if it
>>>>                         doesn't fix your problems, it will be much
>>>>                         easier for us to support.
>>>>
>>>>
>>>>>
>>>>>                         2014-07-09 10:55 GMT+08:00
>>>>>                         <barrykfl at gmail.com
>>>>>                         <mailto:barrykfl at gmail.com>>:
>>>>>
>>>>>                             FYI..
>>>>>                             160: [04/Jul/2014:12:35:30 +0800]
>>>>>                             conn=936207 fd=73 slot=73 connection
>>>>>                             from 192.168.156.89 to 192.168.156.89
>>>>>                             163: [04/Jul/2014:12:35:30 +0800]
>>>>>                             conn=936207 op=-1 fd=73 closed - B1
>>>>>
>>>>>                             There is not abt binding but i unsure
>>>>>                             how to fix ..
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>                             2014-07-09 2:01 GMT+08:00 Rich
>>>>>                             Megginson <rmeggins at redhat.com
>>>>>                             <mailto:rmeggins at redhat.com>>:
>>>>>
>>>>>                                 On 07/08/2014 02:16 AM,
>>>>>                                 barrykfl at gmail.com
>>>>>                                 <mailto:barrykfl at gmail.com> wrote:
>>>>>>                                 Resent as size limit.
>>>>>>
>>>>>>
>>>>>>                                 Here u are  server1 's access log
>>>>>>                                 seem one side broken
>>>>>>
>>>>>>                                 the problem is how to make it
>>>>>>                                 replicate again.
>>>>>>
>>>>>>                                 At server 1
>>>>>>
>>>>>>                                 it is ok  master server1 master
>>>>>>                                 server2
>>>>>>
>>>>>>
>>>>>>                                 Another side server 2 contains 2
>>>>>>                                 ip replication.
>>>>>>
>>>>>>                                 ipa-replica-manage list shown
>>>>>>                                 Can't contact LDAP server
>>>>>>
>>>>>>                                 I dont know why but the
>>>>>>                                 prolematic server is sever 2 not
>>>>>>                                 server 1
>>>>>>
>>>>>>                                 log of server2
>>>>>>                                 [08/Jul/2014:16:02:40 +0800]
>>>>>>                                 conn=3299731 fd=69 slot=69
>>>>>>                                 connection from 192.168.15.89
>>>>>>                                 (server1) to 192.168.15.88(server2)
>>>>>>                                 [08/Jul/2014:16:02:40 +0800]
>>>>>>                                 conn=3299731 op=-1 fd=69 closed - B1
>>>>>>                                 [08/Jul/2014:16:02:40 +0800]
>>>>>>                                 conn=3299732 fd=69 slot=69
>>>>>>                                 connection from 192.168.15.89 to
>>>>>>                                 192.168.15.88
>>>>>>                                 [08/Jul/2014:16:02:40 +0800]
>>>>>>                                 conn=3299732 op=-1 fd=69 closed - B1
>>>>>>                                 [08/Jul/2014:16:02:41 +0800]
>>>>>>                                 conn=3299733 fd=69 slot=69
>>>>>>                                 connection from 192.168.15.89 to
>>>>>>                                 192.168.15.88
>>>>>>                                 [08/Jul/2014:16:02:41 +0800]
>>>>>>                                 conn=3299733 op=-1 fd=69 closed - B1
>>>>>
>>>>>                                 You never answered my question
>>>>>                                 below.  "Are you sure that this
>>>>>                                 connection is a replication
>>>>>                                 session?  Can you post all of the
>>>>>                                 operations from the access log
>>>>>                                 from conn=936207?"
>>>>>
>>>>>                                 In the future, please avoid
>>>>>                                 spamming the list with large log
>>>>>                                 files.  In general, it's better to
>>>>>                                 provide excerpts from the log
>>>>>                                 files showing the problem, paste
>>>>>                                 them to fpaste.org
>>>>>                                 <http://fpaste.org>, and post the
>>>>>                                 link to the mailing list. If for
>>>>>                                 some reason you need to post a
>>>>>                                 large file, please use a file
>>>>>                                 sharing service and post the link
>>>>>                                 to the file.
>>>>>
>>>>>                                 Can you take a look at your errors
>>>>>                                 log from server 1 and server 2 and
>>>>>                                 see if there are any relevant errors?
>>>>>
>>>>>                                 If I had to guess, I would say
>>>>>                                 that there is some sort of network
>>>>>                                 error between server 1 and server
>>>>>                                 2 that causes the excessive closed
>>>>>                                 - B1. Perhaps there will be more
>>>>>                                 information in the errors log.
>>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>                                 2014-07-07 22:21 GMT+08:00 Rich
>>>>>>                                 Megginson <rmeggins at redhat.com
>>>>>>                                 <mailto:rmeggins at redhat.com>>:
>>>>>>
>>>>>>                                     On 07/04/2014 03:28 AM,
>>>>>>                                     barrykfl at gmail.com
>>>>>>                                     <mailto:barrykfl at gmail.com>
>>>>>>                                     wrote:
>>>>>>>                                     FOUND something strange that
>>>>>>>                                     server 1 replicate to itself
>>>>>>>                                     rather than server2
>>>>>>>
>>>>>>>                                     Server1 access log > Wrong
>>>>>>>                                     [04/Jul/2014:12:35:30 +0800]
>>>>>>>                                     conn=936207 fd=73 slot=73
>>>>>>>                                     connection from
>>>>>>>                                     192.168.15.89( server1 )  to
>>>>>>>                                     192.168.15.89 (server1)
>>>>>>
>>>>>>                                     Are you sure that this
>>>>>>                                     connection is a replication
>>>>>>                                     session?  Can you post all of
>>>>>>                                     the operations from the
>>>>>>                                     access log from conn=936207?
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>                                     Server 2 access log > OK
>>>>>>>                                     [04/Jul/2014:12:35:30 +0800]
>>>>>>>                                     conn=936208 fd=74 slot=74
>>>>>>>                                     connection from
>>>>>>>                                     192.168.15.89(server2) to
>>>>>>>                                     192.168.15.88 (server2)
>>>>>>>
>>>>>>>
>>>>>>>                                     2014-07-04 9:25 GMT+08:00
>>>>>>>                                     <barrykfl at gmail.com
>>>>>>>                                     <mailto:barrykfl at gmail.com>>:
>>>>>>>
>>>>>>>                                         Just sure now one side
>>>>>>>                                         flow is broken, if u
>>>>>>>                                         update server1 , it 100%
>>>>>>>                                         work server2 will upgrade.
>>>>>>>                                         but if u update server2
>>>>>>>                                         there is chance non-syn
>>>>>>>                                         e.g it create username
>>>>>>>                                          in server1 with posfix
>>>>>>>                                         grp >ok
>>>>>>>                                         but in server2 it only
>>>>>>>                                         created posfix grp but
>>>>>>>                                         no username /attribute
>>>>>>>                                         it occur serveral times.
>>>>>>>                                         I have to use command
>>>>>>>                                         line grp del ...etc. to
>>>>>>>                                         force del them and
>>>>>>>                                         recreate them.,.
>>>>>>>
>>>>>>>                                         Result below:
>>>>>>>
>>>>>>>                                         server2.abc.com
>>>>>>>                                         <http://server2.abc.com>: replica
>>>>>>>                                           last init status: None
>>>>>>>                                           last init ended: None
>>>>>>>                                           last update status: 0
>>>>>>>                                         Replica acquired
>>>>>>>                                         successfully:
>>>>>>>                                         Incremental update succeeded
>>>>>>>                                           last update ended:
>>>>>>>                                         2014-07-04 00:33:18+00:00
>>>>>>>
>>>>>>>                                         Directory Manager password:
>>>>>>>
>>>>>>>                                         server1.abc.com
>>>>>>>                                         <http://server1.abc.com>: replica
>>>>>>>                                           last init status: 0
>>>>>>>                                         Total update succeeded
>>>>>>>                                           last init ended:
>>>>>>>                                         2014-06-20 10:07:02+00:00
>>>>>>>                                           last update status: 0
>>>>>>>                                         Replica acquired
>>>>>>>                                         successfully:
>>>>>>>                                         Incremental update succeeded
>>>>>>>                                           last update ended:
>>>>>>>                                         2014-07-04 01:14:19+00:00
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>                                         [root@(LIVE)server2 ~]$
>>>>>>>                                          ipactl status
>>>>>>>                                         Directory Service: RUNNING
>>>>>>>                                         KDC Service: RUNNING
>>>>>>>                                         KPASSWD Service: RUNNING
>>>>>>>                                         MEMCACHE Service: RUNNING
>>>>>>>                                         HTTP Service: RUNNING
>>>>>>>
>>>>>>>
>>>>>>>                                         2014-07-04 1:34
>>>>>>>                                         GMT+08:00 Rob Crittenden
>>>>>>>                                         <rcritten at redhat.com
>>>>>>>                                         <mailto:rcritten at redhat.com>>:
>>>>>>>
>>>>>>>
>>>>>>>                                             barrykfl at gmail.com
>>>>>>>                                             <mailto:barrykfl at gmail.com>
>>>>>>>                                             wrote:
>>>>>>>                                             > Yes they are
>>>>>>>                                             running. Server 1
>>>>>>>                                             can syn to server2
>>>>>>>                                             but error at server 2
>>>>>>>                                             > like this.
>>>>>>>
>>>>>>>                                             How do you know
>>>>>>>                                             server 1 is syncing
>>>>>>>                                             with server 2?
>>>>>>>
>>>>>>>                                             On server 1 I'd run:
>>>>>>>
>>>>>>>                                             ipa-replica-manage
>>>>>>>                                             list -v `hostname`
>>>>>>>
>>>>>>>                                             This will show the
>>>>>>>                                             replication status.
>>>>>>>
>>>>>>>                                             And what does ipactl
>>>>>>>                                             status show on server 2?
>>>>>>>
>>>>>>>                                             rob
>>>>>>>
>>>>>>>                                             >
>>>>>>>                                             > 2014/7/3 下午10:14
>>>>>>>                                             於 "Rob Crittenden"
>>>>>>>                                             <rcritten at redhat.com
>>>>>>>                                             <mailto:rcritten at redhat.com>
>>>>>>>                                             >
>>>>>>>                                             <mailto:rcritten at redhat.com
>>>>>>>                                             <mailto:rcritten at redhat.com>>>
>>>>>>>                                             寫道:
>>>>>>>                                             >
>>>>>>>                                             > Please keep relies
>>>>>>>                                             on the list.
>>>>>>>                                             >
>>>>>>>                                             > barrykfl at gmail.com
>>>>>>>                                             <mailto:barrykfl at gmail.com>
>>>>>>>                                             <mailto:barrykfl at gmail.com
>>>>>>>                                             <mailto:barrykfl at gmail.com>>
>>>>>>>                                             wrote:
>>>>>>>                                             >     > I saw the
>>>>>>>                                             error beloe and
>>>>>>>                                             errpr log is it
>>>>>>>                                             related ?
>>>>>>>                                             >     >
>>>>>>>                                             >     >
>>>>>>>                                             29/Jun/2014:02:00:58
>>>>>>>                                             +0800]
>>>>>>>                                             slapd_ldap_sasl_interactive_bind
>>>>>>>                                             - Error:
>>>>>>>                                             >     > could not
>>>>>>>                                             perform interactive
>>>>>>>                                             bind for id [] mech
>>>>>>>                                             [GSSAPI]: LDAP error
>>>>>>>                                             >     > -2 (Local
>>>>>>>                                             error) (SASL(-1):
>>>>>>>                                             generic failure:
>>>>>>>                                             GSSAPI Error:
>>>>>>>                                             Unspecified
>>>>>>>                                             >     > GSS failure.
>>>>>>>                                              Minor code may
>>>>>>>                                             provide more
>>>>>>>                                             information (Credentials
>>>>>>>                                             >     cache
>>>>>>>                                             >     > file
>>>>>>>                                             '/tmp/krb5cc_492'
>>>>>>>                                             not found)) errno 0
>>>>>>>                                             (Success)
>>>>>>>                                             >     >
>>>>>>>                                             [29/Jun/2014:02:00:58 +0800]
>>>>>>>                                             slapi_ldap_bind -
>>>>>>>                                             Error: could not
>>>>>>>                                             > perform
>>>>>>>                                             >     > interactive
>>>>>>>                                             bind for id [] mech
>>>>>>>                                             [GSSAPI]: error -2
>>>>>>>                                             (Local error)
>>>>>>>                                             >
>>>>>>>                                             >     I believe this
>>>>>>>                                             is fairly normal on
>>>>>>>                                             a new startup. It
>>>>>>>                                             has to start
>>>>>>>                                             > somewhere. The
>>>>>>>                                             expired ticket
>>>>>>>                                             errors below are
>>>>>>>                                             unexpected since there
>>>>>>>                                             >     are so many of
>>>>>>>                                             them. Is your KDC
>>>>>>>                                             running?
>>>>>>>                                             >
>>>>>>>                                             > ipactl status
>>>>>>>                                             >
>>>>>>>                                             >     rob
>>>>>>>                                             >
>>>>>>>                                             >     >
>>>>>>>                                             >     >
>>>>>>>                                             >     > 2014-07-02
>>>>>>>                                             14:15 GMT+08:00
>>>>>>>                                             <barrykfl at gmail.com
>>>>>>>                                             <mailto:barrykfl at gmail.com>
>>>>>>>                                             >
>>>>>>>                                             <mailto:barrykfl at gmail.com
>>>>>>>                                             <mailto:barrykfl at gmail.com>>
>>>>>>>                                             <mailto:barrykfl at gmail.com
>>>>>>>                                             <mailto:barrykfl at gmail.com>
>>>>>>>                                             >
>>>>>>>                                             <mailto:barrykfl at gmail.com
>>>>>>>                                             <mailto:barrykfl at gmail.com>>>>:
>>>>>>>                                             >     >
>>>>>>>                                             >     >
>>>>>>>                                             >     >     this is
>>>>>>>                                             the error log i
>>>>>>>                                             found at 2.abc.com
>>>>>>>                                             <http://2.abc.com>
>>>>>>>                                             <http://2.abc.com>
>>>>>>>                                             >     <http://2.abc.com>
>>>>>>>                                             >     >
>>>>>>>                                             >     >
>>>>>>>                                             [30/Jun/2014:12:51:31 +0800]
>>>>>>>                                             slapd_ldap_sasl_interactive_bind
>>>>>>>                                             -
>>>>>>>                                             >     >     Error:
>>>>>>>                                             could not perform
>>>>>>>                                             interactive bind for
>>>>>>>                                             id [] mech [GSSAPI]:
>>>>>>>                                             >     >     LDAP
>>>>>>>                                             error -2 (Local
>>>>>>>                                             error) (SASL(-1):
>>>>>>>                                             generic failure: GSSAPI
>>>>>>>                                             >     >     Error:
>>>>>>>                                             Unspecified GSS
>>>>>>>                                             failure.  Minor code
>>>>>>>                                             may provide more
>>>>>>>                                             >     > information
>>>>>>>                                             (Ticket expired))
>>>>>>>                                             errno 0 (Success)
>>>>>>>                                             >     >
>>>>>>>                                             [30/Jun/2014:12:51:31 +0800]
>>>>>>>                                             slapd_ldap_sasl_interactive_bind
>>>>>>>                                             -
>>>>>>>                                             >     >     Error:
>>>>>>>                                             could not perform
>>>>>>>                                             interactive bind for
>>>>>>>                                             id [] mech [GSSAPI]:
>>>>>>>                                             >     >     LDAP
>>>>>>>                                             error -2 (Local
>>>>>>>                                             error) (SASL(-1):
>>>>>>>                                             generic failure: GSSAPI
>>>>>>>                                             >     >     Error:
>>>>>>>                                             Unspecified GSS
>>>>>>>                                             failure.  Minor code
>>>>>>>                                             may provide more
>>>>>>>                                             >     > information
>>>>>>>                                             (Ticket expired))
>>>>>>>                                             errno 0 (Success)
>>>>>>>                                             >     >
>>>>>>>                                             [30/Jun/2014:12:51:31 +0800]
>>>>>>>                                             slapi_ldap_bind -
>>>>>>>                                             Error: could not
>>>>>>>                                             >     >     perform
>>>>>>>                                             interactive bind for
>>>>>>>                                             id [] mech [GSSAPI]:
>>>>>>>                                             error -2
>>>>>>>                                             > (Local error)
>>>>>>>                                             >     >
>>>>>>>                                             [30/Jun/2014:12:51:31 +0800]
>>>>>>>                                             NSMMReplicationPlugin -
>>>>>>>                                             >     >    
>>>>>>>                                             agmt="cn=meTo1.abc.com
>>>>>>>                                             <http://meTo1.abc.com>
>>>>>>>                                             <http://meTo1.abc.com>
>>>>>>>                                             >    
>>>>>>>                                             <http://meTo1.abc.com>"
>>>>>>>                                             (central:389):
>>>>>>>                                             >     > Replication
>>>>>>>                                             bind with GSSAPI
>>>>>>>                                             auth failed: LDAP
>>>>>>>                                             error -2 (Local
>>>>>>>                                             >     >     error)
>>>>>>>                                             (SASL(-1): generic
>>>>>>>                                             failure: GSSAPI
>>>>>>>                                             Error: Unspecified GSS
>>>>>>>                                             >     >     failure.
>>>>>>>                                              Minor code may
>>>>>>>                                             provide more
>>>>>>>                                             information (Ticket
>>>>>>>                                             > expired))
>>>>>>>                                             >     >
>>>>>>>                                             [30/Jun/2014:12:51:34 +0800]
>>>>>>>                                             slapd_ldap_sasl_interactive_bind
>>>>>>>                                             -
>>>>>>>                                             >     >     Error:
>>>>>>>                                             could not perform
>>>>>>>                                             interactive bind for
>>>>>>>                                             id [] mech [GSSAPI]:
>>>>>>>                                             >     >     LDAP
>>>>>>>                                             error -2 (Local
>>>>>>>                                             error) (SASL(-1):
>>>>>>>                                             generic failure: GSSAPI
>>>>>>>                                             >     >     Error:
>>>>>>>                                             Unspecified GSS
>>>>>>>                                             failure.  Minor code
>>>>>>>                                             may provide more
>>>>>>>                                             >     > information
>>>>>>>                                             (Ticket expired))
>>>>>>>                                             errno 0 (Success)
>>>>>>>                                             >     >
>>>>>>>                                             [30/Jun/2014:12:51:35 +0800]
>>>>>>>                                             slapd_ldap_sasl_interactive_bind
>>>>>>>                                             -
>>>>>>>                                             >     >     Error:
>>>>>>>                                             could not perform
>>>>>>>                                             interactive bind for
>>>>>>>                                             id [] mech [GSSAPI]:
>>>>>>>                                             >     >     LDAP
>>>>>>>                                             error -2 (Local
>>>>>>>                                             error) (SASL(-1):
>>>>>>>                                             generic failure: GSSAPI
>>>>>>>                                             >     >     Error:
>>>>>>>                                             Unspecified GSS
>>>>>>>                                             failure.  Minor code
>>>>>>>                                             may provide more
>>>>>>>                                             >     > information
>>>>>>>                                             (Ticket expired))
>>>>>>>                                             errno 0 (Success)
>>>>>>>                                             >     >
>>>>>>>                                             [30/Jun/2014:12:51:35 +0800]
>>>>>>>                                             slapi_ldap_bind -
>>>>>>>                                             Error: could not
>>>>>>>                                             >     >     perform
>>>>>>>                                             interactive bind for
>>>>>>>                                             id [] mech [GSSAPI]:
>>>>>>>                                             error -2
>>>>>>>                                             > (Local error)
>>>>>>>                                             >     >
>>>>>>>                                             [30/Jun/2014:12:51:40 +0800]
>>>>>>>                                             slapd_ldap_sasl_interactive_bind
>>>>>>>                                             -
>>>>>>>                                             >     >     Error:
>>>>>>>                                             could not perform
>>>>>>>                                             interactive bind for
>>>>>>>                                             id [] mech [GSSAPI]:
>>>>>>>                                             >     >     LDAP
>>>>>>>                                             error -2 (Local
>>>>>>>                                             error) (SASL(-1):
>>>>>>>                                             generic failure: GSSAPI
>>>>>>>                                             >     >     Error:
>>>>>>>                                             Unspecified GSS
>>>>>>>                                             failure.  Minor code
>>>>>>>                                             may provide more
>>>>>>>                                             >     > information
>>>>>>>                                             (Ticket expired))
>>>>>>>                                             errno 0 (Success)
>>>>>>>                                             >     >
>>>>>>>                                             [30/Jun/2014:12:51:40 +0800]
>>>>>>>                                             slapd_ldap_sasl_interactive_bind
>>>>>>>                                             -
>>>>>>>                                             >     >     Error:
>>>>>>>                                             could not perform
>>>>>>>                                             interactive bind for
>>>>>>>                                             id [] mech [GSSAPI]:
>>>>>>>                                             >     >     LDAP
>>>>>>>                                             error -2 (Local
>>>>>>>                                             error) (SASL(-1):
>>>>>>>                                             generic failure: GSSAPI
>>>>>>>                                             >     >     Error:
>>>>>>>                                             Unspecified GSS
>>>>>>>                                             failure.  Minor code
>>>>>>>                                             may provide more
>>>>>>>                                             >     > information
>>>>>>>                                             (Ticket expired))
>>>>>>>                                             errno 0 (Success)
>>>>>>>                                             >     >
>>>>>>>                                             [30/Jun/2014:12:51:40 +0800]
>>>>>>>                                             slapi_ldap_bind -
>>>>>>>                                             Error: could not
>>>>>>>                                             >     >     perform
>>>>>>>                                             interactive bind for
>>>>>>>                                             id [] mech [GSSAPI]:
>>>>>>>                                             error -2
>>>>>>>                                             > (Local error)
>>>>>>>                                             >     >
>>>>>>>                                             >     >
>>>>>>>                                             >     >    
>>>>>>>                                             2014-07-02 12:32
>>>>>>>                                             GMT+08:00
>>>>>>>                                             <barrykfl at gmail.com
>>>>>>>                                             <mailto:barrykfl at gmail.com>
>>>>>>>                                             >
>>>>>>>                                             <mailto:barrykfl at gmail.com
>>>>>>>                                             <mailto:barrykfl at gmail.com>>
>>>>>>>                                             >     >
>>>>>>>                                             <mailto:barrykfl at gmail.com
>>>>>>>                                             <mailto:barrykfl at gmail.com>
>>>>>>>                                             <mailto:barrykfl at gmail.com
>>>>>>>                                             <mailto:barrykfl at gmail.com>>>>:
>>>>>>>                                             > >
>>>>>>>                                             >     >         yes
>>>>>>>                                             on node 1 it is
>>>>>>>                                             happening only node2
>>>>>>>                                             fail connect
>>>>>>>                                             >     >
>>>>>>>                                             >     >
>>>>>>>                                             ipa-replica-manage
>>>>>>>                                             list 2.abc.com
>>>>>>>                                             <http://2.abc.com>
>>>>>>>                                             <http://2.abc.com>
>>>>>>>                                             >     <http://2.abc.com>
>>>>>>>                                             >     > Directory
>>>>>>>                                             Manager password:
>>>>>>>                                             >     >
>>>>>>>                                             >     > 1.abc.com
>>>>>>>                                             <http://1.abc.com>
>>>>>>>                                             <http://1.abc.com>
>>>>>>>                                             <http://1.abc.com>:
>>>>>>>                                             replica
>>>>>>>                                             > >
>>>>>>>                                             >     >
>>>>>>>                                             >     >
>>>>>>>                                             >     > 2014-06-30
>>>>>>>                                             20:59 GMT+08:00 Rob
>>>>>>>                                             Crittenden
>>>>>>>                                             >    
>>>>>>>                                             <rcritten at redhat.com
>>>>>>>                                             <mailto:rcritten at redhat.com>
>>>>>>>                                             <mailto:rcritten at redhat.com
>>>>>>>                                             <mailto:rcritten at redhat.com>>
>>>>>>>                                             >     >
>>>>>>>                                             <mailto:rcritten at redhat.com
>>>>>>>                                             <mailto:rcritten at redhat.com>
>>>>>>>                                             <mailto:rcritten at redhat.com
>>>>>>>                                             <mailto:rcritten at redhat.com>>>>:
>>>>>>>                                             > >
>>>>>>>                                             >     > Barry wrote:
>>>>>>>                                             >     > > Hi:
>>>>>>>                                             >     >
>>>>>>>
>>                 ...
>>
>
>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/freeipa-users/attachments/20140711/2c5d83e2/attachment.htm>


More information about the Freeipa-users mailing list