<div dir="ltr">Hello,<div><br></div><div>I'm running FreeIPA 3 on CentOS 6.8, and have a bit of a bind on my hand. Replication appeared to break with all replicas, and trying to initialize new replicas will not proceed. I've taken my cluster apart to the point where I have one server with no replicas, and attempting to add replicas fails with the response:</div><div><br></div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><font face="monospace, monospace">Update failed! Status: [-2 Total update abortedLDAP error: Local error]</font></blockquote><div><br></div><div>The dirsrv logs on the master show the following error repeating:</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><font face="monospace, monospace">[06/Jan/2017:16:56:10 +0000] NSMMReplicationPlugin - agmt="cn=<a href="http://meToreplica2.example.com">meToreplica2.example.com</a>" (replica2:389): Replica has a different generation ID than the local data. </font></blockquote><div><div class="gmail_signature"><div dir="ltr"><br></div><div>The errors on the replica I'm trying to setup show this errors:</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><font face="monospace, monospace">[06/Jan/2017:16:56:11 +0000] NSMMReplicationPlugin - replica_replace_ruv_tombstone: failed to update replication update vector for replica dc=example,dc=com: LDAP error - 1</font></blockquote><div dir="ltr"><br></div><div>I don't see any other errors in the access or error logs on either the master or replica, and have tried replicating to several new servers, all which consistently fail with the same issue. </div><div dir="ltr"><br></div><div>When running ipa-replica-install in debug mode, the output when things break looks like this:</div><div dir="ltr"><br></div><div dir="ltr"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><font face="monospace, monospace">ipa.ipaserver.plugins.ldap2.SchemaCache: DEBUG retrieving schema for SchemaCache url=ldaps://<a href="http://master.example.com:636">master.example.com:636</a></font> </blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> </blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><font face="monospace, monospace">conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x40d2638><br></font><font face="monospace, monospace">Starting replication, please wait until this has completed.<br></font><font face="monospace, monospace">Update in progress<br></font><font face="monospace, monospace">Update in progress<br></font><font face="monospace, monospace">Update in progress<br></font><font face="monospace, monospace">Update in progress<br></font><font face="monospace, monospace">Update in progress<br></font><font face="monospace, monospace">Update in progress<br></font><font face="monospace, monospace">Update in progress<br></font><font face="monospace, monospace">Update in progress<br></font><font face="monospace, monospace">Update in progress<br></font><font face="monospace, monospace">Update in progress<br></font><font face="monospace, monospace">[</font><span style="font-family:monospace,monospace"><a href="http://master.example.com">master.example.com</a></span><font face="monospace, monospace">] reports: Update failed! Status: [-2 Total update abortedLDAP error: Local error]<br></font><font face="monospace, monospace">ipa : INFO File "/usr/lib/python2.6/site-packages/ipaserver/install/installutils.py", line 614, in run_script<br></font><font face="monospace, monospace"> return_value = main_function()</font><font face="monospace, monospace"><br></font><font face="monospace, monospace"> File "/usr/sbin/ipa-replica-install", line 487, in main<br></font><font face="monospace, monospace"> ds = install_replica_ds(config)</font><font face="monospace, monospace"><br></font><font face="monospace, monospace"> File "/usr/sbin/ipa-replica-install", line 150, in install_replica_ds<br></font><font face="monospace, monospace"> pkcs12_info)</font><font face="monospace, monospace"><br></font><font face="monospace, monospace"> File "/usr/lib/python2.6/site-packages/ipaserver/install/dsinstance.py", line 300, in create_replica<br></font><font face="monospace, monospace"> self.start_creation(runtime=60)</font><font face="monospace, monospace"><br></font><font face="monospace, monospace"> File "/usr/lib/python2.6/site-packages/ipaserver/install/service.py", line 358, in start_creation<br></font><font face="monospace, monospace"> method()</font><font face="monospace, monospace"><br></font><font face="monospace, monospace"> File "/usr/lib/python2.6/site-packages/ipaserver/install/dsinstance.py", line 313, in __setup_replica<br></font><font face="monospace, monospace"> r_bindpw=self.dm_password)</font><font face="monospace, monospace"><br></font><font face="monospace, monospace"> File "/usr/lib/python2.6/site-packages/ipaserver/install/replication.py", line 865, in setup_replication<br></font><font face="monospace, monospace"> raise RuntimeError("Failed to start replication")</font><font face="monospace, monospace"><br></font><font face="monospace, monospace">ipa : INFO The ipa-replica-install command failed, exception: RuntimeError: Failed to start replication</font></blockquote></div><div dir="ltr"><br></div><div>On the master, when tailing the dirsrv access and error logs, the following happens:</div><div dir="ltr"> </div><div dir="ltr"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><font face="monospace, monospace">[06/Jan/2017:19:13:30 +0000] conn=35465 op=16 SRCH base="cn=meTo</font><span style="font-family:monospace,monospace"><a href="http://replica2.example.com">replica2.example.com</a></span><font face="monospace, monospace">,cn=replica,cn=dc\3Dcriticalmention\2Cdc\3Dcom,cn=mapping tree,cn=config" scope=0 filter="(objectClass=*)" attrs="cn nsds5BeginReplicaRefresh nsds5replicaUpdateInProgress nsds5replicaLastInitStatus nsds5replicaLastInitStart nsds5replicaLastInitEnd"<br></font><font face="monospace, monospace">[06/Jan/2017:19:13:30 +0000] conn=35465 op=16 RESULT err=0 tag=101 nentries=1 etime=0<br></font><font face="monospace, monospace">[06/Jan/2017:19:13:30 +0000] conn=35021 op=9 UNBIND<br></font><font face="monospace, monospace">[06/Jan/2017:19:13:30 +0000] conn=35021 op=9 fd=89 closed - U1<br></font><font face="monospace, monospace">[06/Jan/2017:19:13:31 +0000] conn=35465 op=17 SRCH base="cn=meTo</font><span style="font-family:monospace,monospace"><a href="http://replica2.example.com">replica2.example.com</a></span><font face="monospace, monospace">,cn=replica,cn=dc\3Dcriticalmention\2Cdc\3Dcom,cn=mapping tree,cn=config" scope=0 filter="(objectClass=*)" attrs="cn nsds5BeginReplicaRefresh nsds5replicaUpdateInProgress nsds5replicaLastInitStatus nsds5replicaLastInitStart nsds5replicaLastInitEnd"<br></font><font face="monospace, monospace">[06/Jan/2017:19:13:31 +0000] conn=35465 op=17 RESULT err=0 tag=101 nentries=1 etime=0</font> </blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><font face="monospace, monospace">==> /var/log/dirsrv/slapd-EXAMPLE-COM/errors <==<br></font><font face="monospace, monospace">[06/Jan/2017:19:13:37 +0000] NSMMReplicationPlugin - agmt="cn=meTo</font><span style="font-family:monospace,monospace"><a href="http://replica2.example.com">replica2.example.com</a></span><font face="monospace, monospace">" (</font><span style="font-family:monospace,monospace">replica2</span><font face="monospace, monospace">:389): Failed to send extended operation: LDAP error -1 (Can't contact LDAP server)<br></font><font face="monospace, monospace">[06/Jan/2017:19:13:37 +0000] NSMMReplicationPlugin - agmt="cn=meTo</font><span style="font-family:monospace,monospace"><a href="http://replica2.example.com">replica2.example.com</a></span><font face="monospace, monospace">" (</font><span style="font-family:monospace,monospace">replica2</span><font face="monospace, monospace">:389): Received error -1 (Can't contact LDAP server): for total update operation<br></font><font face="monospace, monospace">[06/Jan/2017:19:13:38 +0000] NSMMReplicationPlugin - agmt="cn=meTo</font><span style="font-family:monospace,monospace"><a href="http://replica2.example.com">replica2.example.com</a></span><font face="monospace, monospace">" (</font><span style="font-family:monospace,monospace">replica2</span><font face="monospace, monospace">:389): Warning: unable to send endReplication extended operation (Can't contact LDAP server)<br></font><font face="monospace, monospace">[06/Jan/2017:19:13:38 +0000] NSMMReplicationPlugin - agmt="cn=meTo</font><span style="font-family:monospace,monospace"><a href="http://replica2.example.com">replica2.example.com</a></span><span style="font-family:monospace,monospace">" (</span><span style="font-family:monospace,monospace">replica2</span><span style="font-family:monospace,monospace">:389): Replication bind with SIMPLE auth resumed</span></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><font face="monospace, monospace">[06/Jan/2017:19:13:38 +0000] NSMMReplicationPlugin - agmt="cn=meTo</font><span style="font-family:monospace,monospace"><a href="http://replica2.example.com">replica2.example.com</a></span><font face="monospace, monospace">" (</font><span style="font-family:monospace,monospace">replica2</span><font face="monospace, monospace">:389): Replica has a different generation ID than the local data.</font></blockquote></div><div dir="ltr"><br></div><div>On the replica, in the dirsrv error logs for the same time, the following happens:</div><div><br></div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><font face="monospace, monospace">[06/Jan/2017:19:13:26 +0000] - slapd started. Listening on All Interfaces port 389 for LDAP requests<br>[06/Jan/2017:19:13:26 +0000] - Listening on All Interfaces port 636 for LDAPS requests<br>[06/Jan/2017:19:13:26 +0000] - Listening on /var/run/slapd-EXAMPLE-COM.socket for LDAPI reque<br>sts<br>[06/Jan/2017:19:13:27 +0000] NSMMReplicationPlugin - agmt="cn=<a href="http://meTomaster.example.com">meTomaster.example.com</a>" (master:389): Replica has a different generation ID than the local data.<br>[06/Jan/2017:19:13:27 +0000] NSMMReplicationPlugin - multimaster_be_state_change: replica dc=example,dc=com is going offline; disabling replication<br>[06/Jan/2017:19:13:27 +0000] - WARNING: Import is running with nsslapd-db-private-import-mem on; No other process is allowed to access the database<br>[06/Jan/2017:19:13:28 +0000] - ERROR bulk import abandoned</font></blockquote></div><div dir="ltr"><br></div><div dir="ltr"><br></div><div dir="ltr">Is there something I'm missing which needs to be changed before starting the replication install? The information I found online for different generation ID didn't help me get replication to work, so if there is any advice that could help me, I'd really appreciate it.</div><div dir="ltr"><br></div><div dir="ltr">Thanks a lot<br><div>-- </div><div>Steven Viola</div></div></div></div> </div></div>