[Freeipa-users] Replication issue

Wed Mar 5 11:56:05 UTC 2014

I didn't record the time that the "beaver" user was added to ipa2, but
the logs after the upgrade & reboot are:

ipa01
=====
[04/Mar/2014:19:16:05 +0000] slapd_ldap_sasl_interactive_bind - Error:
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
-1 (Can't contact LDAP server) ((null)) errno 107 (Transport endpoint is
not connected)
[04/Mar/2014:19:16:05 +0000] slapi_ldap_bind - Error: could not perform
interactive bind for id [] mech [GSSAPI]: error -1 (Can't contact LDAP
server)
[04/Mar/2014:19:16:05 +0000] NSMMReplicationPlugin -
agmt="cn=meTolvdlvldap02.unix.vmoney.local" (lvdlvldap02:389):
Replication bind with GSSAPI auth failed: LDAP error -1 (Can't contact
LDAP server) ((null))
[04/Mar/2014:19:16:09 +0000] slapd_ldap_sasl_interactive_bind - Error:
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
-1 (Can't contact LDAP server) ((null)) errno 107 (Transport endpoint is
not connected)
[04/Mar/2014:19:16:09 +0000] slapi_ldap_bind - Error: could not perform
interactive bind for id [] mech [GSSAPI]: error -1 (Can't contact LDAP
server)
[04/Mar/2014:19:16:16 +0000] NSMMReplicationPlugin -
agmt="cn=meTolvdlvldap02.unix.vmoney.local" (lvdlvldap02:389):
Replication bind with GSSAPI auth resumed
[04/Mar/2014:19:26:49 +0000] slapd_ldap_sasl_interactive_bind - Error:
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
-1 (Can't contact LDAP server) ((null)) errno 107 (Transport endpoint is
not connected)
[04/Mar/2014:19:26:49 +0000] slapi_ldap_bind - Error: could not perform
interactive bind for id [] mech [GSSAPI]: error -1 (Can't contact LDAP
server)
[04/Mar/2014:19:26:49 +0000] NSMMReplicationPlugin -
agmt="cn=meTolvdlvldap02.unix.vmoney.local" (lvdlvldap02:389):
Replication bind with GSSAPI auth failed: LDAP error -1 (Can't contact
LDAP server) ((null))
[04/Mar/2014:19:26:55 +0000] slapd_ldap_sasl_interactive_bind - Error:
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
-1 (Can't contact LDAP server) ((null)) errno 107 (Transport endpoint is
not connected)
[04/Mar/2014:19:26:55 +0000] slapi_ldap_bind - Error: could not perform
interactive bind for id [] mech [GSSAPI]: error -1 (Can't contact LDAP
server)
[04/Mar/2014:19:27:01 +0000] slapd_ldap_sasl_interactive_bind - Error:
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
-1 (Can't contact LDAP server) ((null)) errno 107 (Transport endpoint is
not connected)
[04/Mar/2014:19:27:01 +0000] slapi_ldap_bind - Error: could not perform
interactive bind for id [] mech [GSSAPI]: error -1 (Can't contact LDAP
server)
[04/Mar/2014:19:27:13 +0000] slapd_ldap_sasl_interactive_bind - Error:
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
-1 (Can't contact LDAP server) ((null)) errno 107 (Transport endpoint is
not connected)
[04/Mar/2014:19:27:13 +0000] slapi_ldap_bind - Error: could not perform
interactive bind for id [] mech [GSSAPI]: error -1 (Can't contact LDAP
server)
[04/Mar/2014:19:27:37 +0000] slapd_ldap_sasl_interactive_bind - Error:
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
-1 (Can't contact LDAP server) ((null)) errno 107 (Transport endpoint is
not connected)
[04/Mar/2014:19:27:37 +0000] slapi_ldap_bind - Error: could not perform
interactive bind for id [] mech [GSSAPI]: error -1 (Can't contact LDAP
server)
[04/Mar/2014:19:28:25 +0000] slapd_ldap_sasl_interactive_bind - Error:
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
-1 (Can't contact LDAP server) ((null)) errno 107 (Transport endpoint is
not connected)
[04/Mar/2014:19:28:25 +0000] slapi_ldap_bind - Error: could not perform
interactive bind for id [] mech [GSSAPI]: error -1 (Can't contact LDAP
server)
[04/Mar/2014:19:30:01 +0000] slapd_ldap_sasl_interactive_bind - Error:
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
-1 (Can't contact LDAP server) ((null)) errno 107 (Transport endpoint is
not connected)
[04/Mar/2014:19:30:01 +0000] slapi_ldap_bind - Error: could not perform
interactive bind for id [] mech [GSSAPI]: error -1 (Can't contact LDAP
server)
[04/Mar/2014:19:33:13 +0000] slapd_ldap_sasl_interactive_bind - Error:
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
-1 (Can't contact LDAP server) ((null)) errno 107 (Transport endpoint is
not connected)
[04/Mar/2014:19:33:13 +0000] slapi_ldap_bind - Error: could not perform
interactive bind for id [] mech [GSSAPI]: error -1 (Can't contact LDAP
server)
[04/Mar/2014:19:38:13 +0000] slapd_ldap_sasl_interactive_bind - Error:
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
-1 (Can't contact LDAP server) ((null)) errno 107 (Transport endpoint is
not connected)
[04/Mar/2014:19:38:13 +0000] slapi_ldap_bind - Error: could not perform
interactive bind for id [] mech [GSSAPI]: error -1 (Can't contact LDAP
server)
[04/Mar/2014:19:43:13 +0000] slapd_ldap_sasl_interactive_bind - Error:
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
-1 (Can't contact LDAP server) ((null)) errno 107 (Transport endpoint is
not connected)
[04/Mar/2014:19:43:13 +0000] slapi_ldap_bind - Error: could not perform
interactive bind for id [] mech [GSSAPI]: error -1 (Can't contact LDAP
server)
[04/Mar/2014:19:48:10 +0000] NSMMReplicationPlugin -
agmt="cn=meTolvdlvldap02.unix.vmoney.local" (lvdlvldap02:389):
Replication bind with GSSAPI auth resumed
[04/Mar/2014:19:57:08 +0000] - slapd shutting down - signaling operation
threads
[04/Mar/2014:19:57:08 +0000] - slapd shutting down - closing down
internal subsystems and plugins
[04/Mar/2014:19:57:08 +0000] - Waiting for 4 database threads to stop
[04/Mar/2014:19:57:08 +0000] - All database threads now stopped
[04/Mar/2014:19:57:08 +0000] - slapd stopped.
[04/Mar/2014:19:57:44 +0000] - 389-Directory/1.2.11.15 B2013.337.1530
starting up
[04/Mar/2014:19:57:44 +0000] - WARNING: userRoot: entry cache size
10485760B is less than db size 14467072B; We recommend to increase the
entry cache size nsslapd-cachememsize.
[04/Mar/2014:19:57:44 +0000] schema-compat-plugin - warning: no entries
set up under cn=computers, cn=compat,dc=dev,dc=vmoney,dc=local
[04/Mar/2014:19:57:46 +0000] schema-compat-plugin - warning: no entries
set up under ou=sudoers,dc=dev,dc=vmoney,dc=local
[04/Mar/2014:19:57:47 +0000] - Skipping CoS Definition cn=Password
Policy,cn=accounts,dc=dev,dc=vmoney,dc=local--no CoS Templates found,
which should be added before the CoS Definition.
[04/Mar/2014:19:57:47 +0000] - Skipping CoS Definition cn=Password
Policy,cn=accounts,dc=dev,dc=vmoney,dc=local--no CoS Templates found,
which should be added before the CoS Definition.
[04/Mar/2014:19:57:47 +0000] set_krb5_creds - Could not get initial
credentials for principal
[ldap/lvdlvldap01.unix.vmoney.local at DEV.VMONEY.LOCAL] in keytab
[FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for
requested realm)
[04/Mar/2014:19:57:47 +0000] slapd_ldap_sasl_interactive_bind - Error:
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
-2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified
GSS failure.  Minor code may provide more information (Credentials cache
file '/tmp/krb5cc_493' not found)) errno 0 (Success)
[04/Mar/2014:19:57:47 +0000] slapi_ldap_bind - Error: could not perform
interactive bind for id [] mech [GSSAPI]: error -2 (Local error)
[04/Mar/2014:19:57:47 +0000] NSMMReplicationPlugin -
agmt="cn=meTolvdlvldap02.unix.vmoney.local" (lvdlvldap02:389):
Replication bind with GSSAPI auth failed: LDAP error -2 (Local error)
(SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.
Minor code may provide more information (Credentials cache file
'/tmp/krb5cc_493' not found))
[04/Mar/2014:19:57:47 +0000] - slapd started.  Listening on All
Interfaces port 389 for LDAP requests
[04/Mar/2014:19:57:47 +0000] - Listening on All Interfaces port 636 for
LDAPS requests
[04/Mar/2014:19:57:47 +0000] - Listening on
/var/run/slapd-DEV-VMONEY-LOCAL.socket for LDAPI requests
[04/Mar/2014:19:57:51 +0000] NSMMReplicationPlugin -
agmt="cn=meTolvdlvldap02.unix.vmoney.local" (lvdlvldap02:389):
Replication bind with GSSAPI auth resumed

ipa02
=====
[04/Mar/2014:19:16:07 +0000] - 389-Directory/1.2.11.15 B2013.337.1530
starting up
[04/Mar/2014:19:16:08 +0000] - WARNING: userRoot: entry cache size
10485760B is less than db size 14401536B; We recommend to increase the
entry cache size nsslapd-cachememsize.
[04/Mar/2014:19:16:08 +0000] schema-compat-plugin - warning: no entries
set up under cn=computers, cn=compat,dc=dev,dc=vmoney,dc=local
[04/Mar/2014:19:16:10 +0000] schema-compat-plugin - warning: no entries
set up under ou=sudoers,dc=dev,dc=vmoney,dc=local
[04/Mar/2014:19:16:11 +0000] - Skipping CoS Definition cn=Password
Policy,cn=accounts,dc=dev,dc=vmoney,dc=local--no CoS Templates found,
which should be added before the CoS Definition.
[04/Mar/2014:19:16:11 +0000] - Skipping CoS Definition cn=Password
Policy,cn=accounts,dc=dev,dc=vmoney,dc=local--no CoS Templates found,
which should be added before the CoS Definition.
[04/Mar/2014:19:16:11 +0000] set_krb5_creds - Could not get initial
credentials for principal
[ldap/lvdlvldap02.unix.vmoney.local at DEV.VMONEY.LOCAL] in keytab
[FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for
requested realm)
[04/Mar/2014:19:16:11 +0000] slapd_ldap_sasl_interactive_bind - Error:
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
-2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified
GSS failure.  Minor code may provide more information (Credentials cache
file '/tmp/krb5cc_495' not found)) errno 0 (Success)
[04/Mar/2014:19:16:11 +0000] slapi_ldap_bind - Error: could not perform
interactive bind for id [] mech [GSSAPI]: error -2 (Local error)
[04/Mar/2014:19:16:11 +0000] NSMMReplicationPlugin -
agmt="cn=meTolvdlvldap01.unix.vmoney.local" (lvdlvldap01:389):
Replication bind with GSSAPI auth failed: LDAP error -2 (Local error)
(SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.
Minor code may provide more information (Credentials cache file
'/tmp/krb5cc_495' not found))
[04/Mar/2014:19:16:11 +0000] - slapd started.  Listening on All
Interfaces port 389 for LDAP requests
[04/Mar/2014:19:16:11 +0000] - Listening on All Interfaces port 636 for
LDAPS requests
[04/Mar/2014:19:16:11 +0000] - Listening on
/var/run/slapd-DEV-VMONEY-LOCAL.socket for LDAPI requests
[04/Mar/2014:19:16:14 +0000] NSMMReplicationPlugin -
agmt="cn=meTolvdlvldap01.unix.vmoney.local" (lvdlvldap01:389):
Replication bind with GSSAPI auth resumed
[04/Mar/2014:19:22:07 +0000] - slapd shutting down - signaling operation
threads
[04/Mar/2014:19:22:07 +0000] - slapd shutting down - closing down
internal subsystems and plugins
[04/Mar/2014:19:22:08 +0000] - Waiting for 4 database threads to stop
[04/Mar/2014:19:22:08 +0000] - All database threads now stopped
[04/Mar/2014:19:22:08 +0000] - slapd stopped.
[04/Mar/2014:19:47:32 +0000] - 389-Directory/1.2.11.15 B2013.337.1530
starting up
[04/Mar/2014:19:47:32 +0000] - WARNING: userRoot: entry cache size
10485760B is less than db size 14401536B; We recommend to increase the
entry cache size nsslapd-cachememsize.
[04/Mar/2014:19:47:32 +0000] schema-compat-plugin - warning: no entries
set up under cn=computers, cn=compat,dc=dev,dc=vmoney,dc=local
[04/Mar/2014:19:47:34 +0000] schema-compat-plugin - warning: no entries
set up under ou=sudoers,dc=dev,dc=vmoney,dc=local
[04/Mar/2014:19:47:35 +0000] - Skipping CoS Definition cn=Password
Policy,cn=accounts,dc=dev,dc=vmoney,dc=local--no CoS Templates found,
which should be added before the CoS Definition.
[04/Mar/2014:19:47:35 +0000] - Skipping CoS Definition cn=Password
Policy,cn=accounts,dc=dev,dc=vmoney,dc=local--no CoS Templates found,
which should be added before the CoS Definition.
[04/Mar/2014:19:47:35 +0000] set_krb5_creds - Could not get initial
credentials for principal
[ldap/lvdlvldap02.unix.vmoney.local at DEV.VMONEY.LOCAL] in keytab
[FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for
requested realm)
[04/Mar/2014:19:47:35 +0000] slapd_ldap_sasl_interactive_bind - Error:
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
-2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified
GSS failure.  Minor code may provide more information (Credentials cache
file '/tmp/krb5cc_495' not found)) errno 0 (Success)
[04/Mar/2014:19:47:35 +0000] slapi_ldap_bind - Error: could not perform
interactive bind for id [] mech [GSSAPI]: error -2 (Local error)
[04/Mar/2014:19:47:35 +0000] NSMMReplicationPlugin -
agmt="cn=meTolvdlvldap01.unix.vmoney.local" (lvdlvldap01:389):
Replication bind with GSSAPI auth failed: LDAP error -2 (Local error)
(SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.
Minor code may provide more information (Credentials cache file
'/tmp/krb5cc_495' not found))
[04/Mar/2014:19:47:35 +0000] - slapd started.  Listening on All
Interfaces port 389 for LDAP requests
[04/Mar/2014:19:47:35 +0000] - Listening on All Interfaces port 636 for
LDAPS requests
[04/Mar/2014:19:47:35 +0000] - Listening on
/var/run/slapd-DEV-VMONEY-LOCAL.socket for LDAPI requests
[04/Mar/2014:19:47:39 +0000] NSMMReplicationPlugin -
agmt="cn=meTolvdlvldap01.unix.vmoney.local" (lvdlvldap01:389):
Replication bind with GSSAPI auth resumed
[04/Mar/2014:19:54:10 +0000] NSMMReplicationPlugin - changelog program -
_cl5WriteOperationTxn: retry (49) the transaction
(csn=53162f5f000000030000) failed (rc=-30994 (DB_LOCK_DEADLOCK: Locker
killed to resolve a deadlock))
[04/Mar/2014:19:54:10 +0000] NSMMReplicationPlugin - changelog program -
_cl5WriteOperationTxn: failed to write entry with csn
(53162f5f000000030000); db error - -30994 DB_LOCK_DEADLOCK: Locker
killed to resolve a deadlock
[04/Mar/2014:19:54:10 +0000] NSMMReplicationPlugin -
write_changelog_and_ruv: can't add a change for
uid=beaver,cn=users,cn=accounts,dc=dev,dc=vmoney,dc=local (uniqid:
a9e60601-a3d611e3-ba5495ee-66868ebf, optype: 16) to changelog csn
53162f5f000000030000
[04/Mar/2014:19:59:38 +0000] - slapd shutting down - signaling operation
threads
[04/Mar/2014:19:59:38 +0000] - slapd shutting down - closing down
internal subsystems and plugins
[04/Mar/2014:19:59:38 +0000] - Waiting for 4 database threads to stop
[04/Mar/2014:19:59:39 +0000] - All database threads now stopped
[04/Mar/2014:19:59:39 +0000] - slapd stopped.
[04/Mar/2014:20:00:16 +0000] - 389-Directory/1.2.11.15 B2013.337.1530
starting up
[04/Mar/2014:20:00:16 +0000] - WARNING: userRoot: entry cache size
10485760B is less than db size 14434304B; We recommend to increase the
entry cache size nsslapd-cachememsize.
[04/Mar/2014:20:00:16 +0000] schema-compat-plugin - warning: no entries
set up under cn=computers, cn=compat,dc=dev,dc=vmoney,dc=local
[04/Mar/2014:20:00:18 +0000] schema-compat-plugin - warning: no entries
set up under ou=sudoers,dc=dev,dc=vmoney,dc=local
[04/Mar/2014:20:00:18 +0000] - Skipping CoS Definition cn=Password
Policy,cn=accounts,dc=dev,dc=vmoney,dc=local--no CoS Templates found,
which should be added before the CoS Definition.
[04/Mar/2014:20:00:19 +0000] - Skipping CoS Definition cn=Password
Policy,cn=accounts,dc=dev,dc=vmoney,dc=local--no CoS Templates found,
which should be added before the CoS Definition.
[04/Mar/2014:20:00:19 +0000] set_krb5_creds - Could not get initial
credentials for principal
[ldap/lvdlvldap02.unix.vmoney.local at DEV.VMONEY.LOCAL] in keytab
[FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for
requested realm)
[04/Mar/2014:20:00:19 +0000] slapd_ldap_sasl_interactive_bind - Error:
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
-2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified
GSS failure.  Minor code may provide more information (Credentials cache
file '/tmp/krb5cc_495' not found)) errno 0 (Success)
[04/Mar/2014:20:00:19 +0000] slapi_ldap_bind - Error: could not perform
interactive bind for id [] mech [GSSAPI]: error -2 (Local error)
[04/Mar/2014:20:00:19 +0000] NSMMReplicationPlugin -
agmt="cn=meTolvdlvldap01.unix.vmoney.local" (lvdlvldap01:389):
Replication bind with GSSAPI auth failed: LDAP error -2 (Local error)
(SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.
Minor code may provide more information (Credentials cache file
'/tmp/krb5cc_495' not found))
[04/Mar/2014:20:00:19 +0000] - slapd started.  Listening on All
Interfaces port 389 for LDAP requests
[04/Mar/2014:20:00:19 +0000] - Listening on All Interfaces port 636 for
LDAPS requests
[04/Mar/2014:20:00:19 +0000] - Listening on
/var/run/slapd-DEV-VMONEY-LOCAL.socket for LDAPI requests
[04/Mar/2014:20:00:22 +0000] NSMMReplicationPlugin -
agmt="cn=meTolvdlvldap01.unix.vmoney.local" (lvdlvldap01:389):
Replication bind with GSSAPI auth resumed

The confusing point for me is that users were successfully added in each
direction before and after the failing "beaver" user.

Cheers
Duncan

________________________________

	From: Rich Megginson [mailto:rmeggins at redhat.com] 
	Sent: 04 March 2014 22:41
	To: Innes, Duncan; freeipa-users at redhat.com
	Subject: Re: [Freeipa-users] Replication issue

	On 03/04/2014 01:22 PM, Innes, Duncan wrote:

		Hi,

		I'm testing an upgrade of my prod IPA servers in a dev
cluster at the moment.  Finally completed the upgrade, so I tested some
user adds via the WebUI.

		Added user "aardvark" on ipa01 - replicated to ipa02
		Added user "beaver" on ipa02 - NOT replicated to ipa01
		Added user "banana" on ipa02 - replicated to ipa01
		Added user "elephant" on ipa02 - replicated to ipa01
		Edited user "beaver" on ipa02 - NOT replicated to ipa01

	Is there anything in /var/log/dirsrv/slapd-DOMAIN-COM/errors on
ipa01 or ipa02?

		Is there anything I can do to force IPA to replicate
that user from ipa02 to ipa01?

		I have tried running 'ipa-replica-manage force-sync
--from ipa02' on ipa01, but it hasn't appeared to do anything.

		Thanks

		Duncan

		This message has been checked for viruses and spam by
the Virgin Money email scanning system powered by Messagelabs.

		This e-mail is intended to be confidential to the
recipient. If you receive a copy in error, please inform the sender and
then delete this message.

		Virgin Money plc - Registered in England and Wales
(Company no. 6952311). Registered office - Jubilee House, Gosforth,
Newcastle upon Tyne NE3 4PL. Virgin Money plc is authorised by the
Prudential Regulation Authority and regulated by the Financial Conduct
Authority and the Prudential Regulation Authority.

		The following companies also trade as Virgin Money. They
are both authorised and regulated by the Financial Conduct Authority,
are registered in England and Wales and have their registered office at
Jubilee House, Gosforth, Newcastle upon Tyne NE3 4PL: Virgin Money
Personal Financial Service Limited (Company no. 3072766) and Virgin
Money Unit Trust Managers Limited (Company no. 3000482).

		For further details of Virgin Money group companies
please visit our website at virginmoney.com

		_______________________________________________
		Freeipa-users mailing list
		Freeipa-users at redhat.com
		https://www.redhat.com/mailman/listinfo/freeipa-users

	This message has been checked for viruses and spam by the Virgin
Money email scanning system powered by Messagelabs.

This message has been checked for viruses and spam by the Virgin Money email scanning system powered by Messagelabs.

This e-mail is intended to be confidential to the recipient. If you receive a copy in error, please inform the sender and then delete this message.

Virgin Money plc - Registered in England and Wales (Company no. 6952311). Registered office - Jubilee House, Gosforth, Newcastle upon Tyne NE3 4PL. Virgin Money plc is authorised by the Prudential Regulation Authority and regulated by the Financial Conduct Authority and the Prudential Regulation Authority.

The following companies also trade as Virgin Money. They are both authorised and regulated by the Financial Conduct Authority, are registered in England and Wales and have their registered office at Jubilee House, Gosforth, Newcastle upon Tyne NE3 4PL: Virgin Money Personal Financial Service Limited (Company no. 3072766) and Virgin Money Unit Trust Managers Limited (Company no. 3000482).

For further details of Virgin Money group companies please visit our website at virginmoney.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/freeipa-users/attachments/20140305/bc4f1b47/attachment.htm>