[Freeipa-users] Unable to add new replicas - Total update aborted - Replica has a different generation ID than the local data.

Fri Jan 6 20:12:07 UTC 2017

Hello,

I'm running FreeIPA 3 on CentOS 6.8, and have a bit of a bind on my hand.
Replication appeared to break with all replicas, and trying to initialize
new replicas will not proceed. I've taken my cluster apart to the point
where I have one server with no replicas, and attempting to add replicas
fails with the response:

Update failed! Status: [-2 Total update abortedLDAP error: Local error]

The dirsrv logs on the master show the following error repeating:

[06/Jan/2017:16:56:10 +0000] NSMMReplicationPlugin - agmt="cn=
> meToreplica2.example.com" (replica2:389): Replica has a different
> generation ID than the local data.

The errors on the replica I'm trying to setup show this errors:

[06/Jan/2017:16:56:11 +0000] NSMMReplicationPlugin -
> replica_replace_ruv_tombstone: failed to update replication update vector
> for replica dc=example,dc=com: LDAP error - 1

I don't see any other errors in the access or error logs on either the
master or replica, and have tried replicating to several new servers, all
which consistently fail with the same issue.

When running ipa-replica-install in debug mode, the output when things
break looks like this:

ipa.ipaserver.plugins.ldap2.SchemaCache: DEBUG    retrieving schema for
> SchemaCache url=ldaps://master.example.com:636

conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x40d2638>
> Starting replication, please wait until this has completed.
> Update in progress
> Update in progress
> Update in progress
> Update in progress
> Update in progress
> Update in progress
> Update in progress
> Update in progress
> Update in progress
> Update in progress
> [master.example.com] reports: Update failed! Status: [-2 Total update
> abortedLDAP error: Local error]
> ipa         : INFO       File
> "/usr/lib/python2.6/site-packages/ipaserver/install/installutils.py", line
> 614, in run_script
>     return_value = main_function()
>   File "/usr/sbin/ipa-replica-install", line 487, in main
>     ds = install_replica_ds(config)
>   File "/usr/sbin/ipa-replica-install", line 150, in install_replica_ds
>     pkcs12_info)
>   File "/usr/lib/python2.6/site-packages/ipaserver/install/dsinstance.py",
> line 300, in create_replica
>     self.start_creation(runtime=60)
>   File "/usr/lib/python2.6/site-packages/ipaserver/install/service.py",
> line 358, in start_creation
>     method()
>   File "/usr/lib/python2.6/site-packages/ipaserver/install/dsinstance.py",
> line 313, in __setup_replica
>     r_bindpw=self.dm_password)
>   File
> "/usr/lib/python2.6/site-packages/ipaserver/install/replication.py", line
> 865, in setup_replication
>     raise RuntimeError("Failed to start replication")
> ipa         : INFO     The ipa-replica-install command failed, exception:
> RuntimeError: Failed to start replication

On the master, when tailing the dirsrv access and error logs, the following
happens:

> [06/Jan/2017:19:13:30 +0000] conn=35465 op=16 SRCH base="cn=meTo
> replica2.example.com,cn=replica,cn=dc\3Dcriticalmention\2Cdc\3Dcom,cn=mapping
> tree,cn=config" scope=0 filter="(objectClass=*)" attrs="cn
> nsds5BeginReplicaRefresh nsds5replicaUpdateInProgress
> nsds5replicaLastInitStatus nsds5replicaLastInitStart
> nsds5replicaLastInitEnd"
> [06/Jan/2017:19:13:30 +0000] conn=35465 op=16 RESULT err=0 tag=101
> nentries=1 etime=0
> [06/Jan/2017:19:13:30 +0000] conn=35021 op=9 UNBIND
> [06/Jan/2017:19:13:30 +0000] conn=35021 op=9 fd=89 closed - U1
> [06/Jan/2017:19:13:31 +0000] conn=35465 op=17 SRCH base="cn=meTo
> replica2.example.com,cn=replica,cn=dc\3Dcriticalmention\2Cdc\3Dcom,cn=mapping
> tree,cn=config" scope=0 filter="(objectClass=*)" attrs="cn
> nsds5BeginReplicaRefresh nsds5replicaUpdateInProgress
> nsds5replicaLastInitStatus nsds5replicaLastInitStart
> nsds5replicaLastInitEnd"
> [06/Jan/2017:19:13:31 +0000] conn=35465 op=17 RESULT err=0 tag=101
> nentries=1 etime=0

==> /var/log/dirsrv/slapd-EXAMPLE-COM/errors <==
> [06/Jan/2017:19:13:37 +0000] NSMMReplicationPlugin - agmt="cn=meTo
> replica2.example.com" (replica2:389): Failed to send extended operation:
> LDAP error -1 (Can't contact LDAP server)
> [06/Jan/2017:19:13:37 +0000] NSMMReplicationPlugin - agmt="cn=meTo
> replica2.example.com" (replica2:389): Received error -1 (Can't contact
> LDAP server):  for total update operation
> [06/Jan/2017:19:13:38 +0000] NSMMReplicationPlugin - agmt="cn=meTo
> replica2.example.com" (replica2:389): Warning: unable to send
> endReplication extended operation (Can't contact LDAP server)
> [06/Jan/2017:19:13:38 +0000] NSMMReplicationPlugin - agmt="cn=meTo
> replica2.example.com" (replica2:389): Replication bind with SIMPLE auth
> resumed

[06/Jan/2017:19:13:38 +0000] NSMMReplicationPlugin - agmt="cn=meTo
> replica2.example.com" (replica2:389): Replica has a different generation
> ID than the local data.

On the replica, in the dirsrv error logs for the same time, the following
happens:

[06/Jan/2017:19:13:26 +0000] - slapd started.  Listening on All Interfaces
> port 389 for LDAP requests
> [06/Jan/2017:19:13:26 +0000] - Listening on All Interfaces port 636 for
> LDAPS requests
> [06/Jan/2017:19:13:26 +0000] - Listening on
> /var/run/slapd-EXAMPLE-COM.socket for LDAPI reque
> sts
> [06/Jan/2017:19:13:27 +0000] NSMMReplicationPlugin - agmt="cn=
> meTomaster.example.com" (master:389): Replica has a different generation
> ID than the local data.
> [06/Jan/2017:19:13:27 +0000] NSMMReplicationPlugin -
> multimaster_be_state_change: replica dc=example,dc=com is going offline;
> disabling replication
> [06/Jan/2017:19:13:27 +0000] - WARNING: Import is running with
> nsslapd-db-private-import-mem on; No other process is allowed to access the
> database
> [06/Jan/2017:19:13:28 +0000] - ERROR bulk import abandoned

Is there something I'm missing which needs to be changed before starting
the replication install? The information I found online for different
generation ID didn't help me get replication to work, so if there is any
advice that could help me, I'd really appreciate it.

Thanks a lot
-- 
Steven Viola
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/freeipa-users/attachments/20170106/0cd8a781/attachment.htm>