[Freeipa-users] Replication failing on FreeIPA 4.2.0

Tue Jan 12 04:30:46 UTC 2016

I have 3 FreeIPA 4.2.0 servers running on CentOS 7.2

I am getting replication errors that I cannot seem to figure out.

Here is the setup : (I refer to master and slave because apparently your
CA is the only one who can create replica certs so it is the 'master')

dc1 : master, been running for a long time on 4.1.4, recently upgraded to
4.2.0
dc2 : replica, been running for a long time on 4.1.4, recently upgraded to
4.2.0
dc3 : replica, newly added as fresh freeipa 4.2.0 after the other 2 were
upgraded.

Changes from dc2 were not being replicated to dc1 for a long time and I
had to ipa-replica-manage re-initialize 3 times for it to finally start
replicating again.  Every time it reported success, but the first 2 times,
any changes on dc2 were not replicated to dc1.

Although replication seems to be working again, I've not got a bunch of
errors in my logs and status checks, and fear it may start failing in the
future again due to some verbage in the log entries.

Also, although I've read the busy replica error is supposed to be
'transient' i've been refreshing the output of the replica-manage list
command for an hour and it hasn't gone away...

I'm also quite confused about the 1970 dates...

[root at dc1 slapd-MYDOMAIN-NET]# ipa-replica-manage list -v `hostname`
dc2.mydomain.net: replica
  last init status: 0 Total update succeeded
  last init ended: 2016-01-12 04:08:47+00:00
  last update status: 0 Replica acquired successfully: Incremental update
succeeded
  last update ended: 2016-01-12 04:25:15+00:00
dc3.mydomain.net: replica
  last init status: 0 Total update succeeded
  last init ended: 2016-01-10 08:06:35+00:00
  last update status: 0 Replica acquired successfully: Incremental update
succeeded
  last update ended: 2016-01-12 04:25:15+00:00

[root at dc2 slapd-MYDOMAIN-NET]# ipa-replica-manage list -v `hostname`
dc1.mydomain.net: replica
  last init status: 1 Replication error acquiring replica: replica busy
  last init ended: 1970-01-01 00:00:00+00:00
  last update status: 1 Can't acquire busy replica
  last update ended: 2016-01-12 04:25:05+00:00

  [root at dc3 slapd-MYDOMAIN-NET]# ipa-replica-manage list -v `hostname`
dc1.mydomain.net: replica
  last init status: None
  last init ended: 1970-01-01 00:00:00+00:00
  last update status: 0 Replica acquired successfully: Incremental update
started
  last update ended: 1970-01-01 00:00:00+00:00

dc2 error logs :
----------------
[12/Jan/2016:04:08:47 +0000] NSMMReplicationPlugin - replica_reload_ruv:
Warning: new data for replica dc=mycompany,dc=net does not match the data
in the changelog.
 Recreating the changelog file. This could affect replication with
replica's  consumers in which case the consumers should be reinitialized.
[12/Jan/2016:04:08:47 +0000] NSMMReplicationPlugin - replication keep
alive entry <cn=repl keep alive 3,dc=mycompany,dc=net> already exists
[12/Jan/2016:04:08:47 +0000] NSACLPlugin - The ACL target
cn=vaults,cn=kra,dc=mycompany,dc=net does not exist
[12/Jan/2016:04:08:47 +0000] NSACLPlugin - The ACL target
cn=vaults,cn=kra,dc=mycompany,dc=net does not exist
[12/Jan/2016:04:08:47 +0000] NSACLPlugin - The ACL target
cn=vaults,cn=kra,dc=mycompany,dc=net does not exist
[12/Jan/2016:04:08:47 +0000] NSACLPlugin - The ACL target
cn=vaults,cn=kra,dc=mycompany,dc=net does not exist
[12/Jan/2016:04:08:47 +0000] NSACLPlugin - The ACL target
cn=vaults,cn=kra,dc=mycompany,dc=net does not exist
[12/Jan/2016:04:08:47 +0000] NSACLPlugin - The ACL target
cn=vaults,cn=kra,dc=mycompany,dc=net does not exist
[12/Jan/2016:04:08:47 +0000] NSACLPlugin - The ACL target
cn=vaults,cn=kra,dc=mycompany,dc=net does not exist
[12/Jan/2016:04:08:47 +0000] NSACLPlugin - The ACL target
cn=vaults,cn=kra,dc=mycompany,dc=net does not exist
[12/Jan/2016:04:08:47 +0000] NSACLPlugin - The ACL target
cn=vaults,cn=kra,dc=mycompany,dc=net does not exist
[12/Jan/2016:04:08:47 +0000] NSACLPlugin - The ACL target
cn=vaults,cn=kra,dc=mycompany,dc=net does not exist
[12/Jan/2016:04:08:47 +0000] NSACLPlugin - The ACL target
cn=vaults,cn=kra,dc=mycompany,dc=net does not exist
[12/Jan/2016:04:08:47 +0000] NSACLPlugin - The ACL target cn=casigningcert
cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=mycompany,dc=net does not exist
[12/Jan/2016:04:08:47 +0000] NSACLPlugin - The ACL target cn=casigningcert
cert-pki-ca,cn=ca_renewal,cn=ipa,cn=etc,dc=mycompany,dc=net does not exist
[12/Jan/2016:04:09:46 +0000] agmt="cn=meTodc1.mycompany.net" (dc1:389) -
Can't locate CSN 56947cbe000800030000 in the changelog (DB rc=-30988). If
replication stops, the consumer may need to be reinitialized.

dc1 error logs :
----------------
[12/Jan/2016:04:08:07 +0000] NSMMReplicationPlugin - Beginning total
update of replica "agmt="cn=meTodc2.mycompany.net" (dc2:389)".
[12/Jan/2016:04:08:07 +0000] NSMMReplicationPlugin - replication keep
alive entry <cn=repl keep alive 4,dc=mycompany,dc=net> already exists
[12/Jan/2016:04:08:48 +0000] NSMMReplicationPlugin - Finished total update
of replica "agmt="cn=meTodc2.mycompany.net" (dc2:389)". Sent 7700 entries.
[12/Jan/2016:04:09:34 +0000] NSMMReplicationPlugin - replication keep
alive entry <cn=repl keep alive 4,dc=mycompany,dc=net> already exists
[12/Jan/2016:04:14:17 +0000] NSMMReplicationPlugin - replication keep
alive entry <cn=repl keep alive 4,dc=mycompany,dc=net> already exists
[12/Jan/2016:04:14:17 +0000] NSMMReplicationPlugin - replication keep
alive entry <cn=repl keep alive 4,dc=mycompany,dc=net> already exists
[12/Jan/2016:04:18:58 +0000] NSMMReplicationPlugin - replication keep
alive entry <cn=repl keep alive 4,dc=mycompany,dc=net> already exists
[12/Jan/2016:04:18:58 +0000] NSMMReplicationPlugin - replication keep
alive entry <cn=repl keep alive 4,dc=mycompany,dc=net> already exists

dc3 error logs :
----------------
[12/Jan/2016:02:24:34 +0000] NSMMReplicationPlugin - replication keep
alive entry <cn=repl keep alive 5,dc=mycompany,dc=net> already exists
[12/Jan/2016:03:05:13 +0000] NSMMReplicationPlugin - replication keep
alive entry <cn=repl keep alive 5,dc=mycompany,dc=net> already exists
[12/Jan/2016:04:03:59 +0000] NSMMReplicationPlugin - replication keep
alive entry <cn=repl keep alive 5,dc=mycompany,dc=net> already exists
[12/Jan/2016:04:08:35 +0000] NSMMReplicationPlugin - replication keep
alive entry <cn=repl keep alive 5,dc=mycompany,dc=net> already exists
[12/Jan/2016:04:14:02 +0000] NSMMReplicationPlugin - replication keep
alive entry <cn=repl keep alive 5,dc=mycompany,dc=net> already exists
[12/Jan/2016:04:20:23 +0000] NSMMReplicationPlugin - replication keep
alive entry <cn=repl keep alive 5,dc=mycompany,dc=net> already exists