[Freeipa-users] IPA Replication Status

[Martin Kosek]

On 07/23/2014 01:58 PM, Choudhury, Suhail wrote:
> I have the following errors on different boxes:
> 
> [root at recsds1 sch32]# tail -f /var/log/dirsrv/slapd-RECS-BSKYB-COM/errors
> [23/Jul/2014:12:28:54 +0100] NSMMReplicationPlugin - CleanAllRUV Task: Replicas have not been cleaned yet, retrying in 10 seconds
> [23/Jul/2014:12:29:06 +0100] NSMMReplicationPlugin - CleanAllRUV Task: Waiting for all the replicas to finish cleaning...
> [23/Jul/2014:12:29:06 +0100] NSMMReplicationPlugin - CleanAllRUV Task: Not all replicas finished cleaning, retrying in 10 seconds
> [23/Jul/2014:12:29:16 +0100] NSMMReplicationPlugin - CleanAllRUV Task: Not all replicas finished cleaning, retrying in 20 seconds
> [23/Jul/2014:12:29:36 +0100] NSMMReplicationPlugin - CleanAllRUV Task: Not all replicas finished cleaning, retrying in 40 seconds
> 
> [root at recsds3 sch32]# tail -f /var/log/dirsrv/slapd-RECS-BSKYB-COM/errors
> [23/Jul/2014:12:52:10 +0100] agmt="cn=meTorecsds2.bskyb.com" (recsds2:389) - Can't locate CSN 53c7ba27000000100000 in the changelog (DB rc=-30988). The consumer may need to be reinitialized.
> [23/Jul/2014:12:52:10 +0100] NSMMReplicationPlugin - agmt="cn=meTorecsds2.bskyb.com" (recsds2:389): changelog iteration code returned a dummy entry with csn 53c7c6b1000200100000, skipping ...
> [23/Jul/2014:12:52:13 +0100] agmt="cn=meTorecsds4.bskyb.com" (recsds4:389) - Can't locate CSN 53c7ba75000400100000 in the changelog (DB rc=-30988). The consumer may need to be reinitialized.
> [23/Jul/2014:12:52:13 +0100] NSMMReplicationPlugin - agmt="cn=meTorecsds4.bskyb.com" (recsds4:389): changelog iteration code returned a dummy entry with csn 53c7c6b1000200100000, skipping ...
> [23/Jul/2014:12:52:13 +0100] agmt="cn=meTorecsds2.bskyb.com" (recsds2:389) - Can't locate CSN 53c7ba27000000100000 in the changelog (DB rc=-30988). The consumer may need to be reinitialized.
> 
> [root at recsds4 ~]# tail -f /var/log/dirsrv/slapd-RECS-BSKYB-COM/errors
> [23/Jul/2014:12:52:03 +0100] ldbm_back_modify - Attempt to modify a tombstone entry nsuniqueid=b0838195-0da911e4-9433f833-313b8581,krbprincipalname=DNS/recsds1.bskyb.com at RECS.BSKYB.COM,cn=services,cn=accounts,dc=recs,dc=bskyb,dc=com
> [23/Jul/2014:12:52:03 +0100] ldbm_back_modify - Attempt to modify a tombstone entry nsuniqueid=85992d8b-0da911e4-9433f833-313b8581,fqdn=recsds1.bskyb.com,cn=computers,cn=accounts,dc=recs,dc=bskyb,dc=com
> [23/Jul/2014:12:52:06 +0100] ldbm_back_modify - Attempt to modify a tombstone entry nsuniqueid=b0838195-0da911e4-9433f833-313b8581,krbprincipalname=DNS/recsds1.bskyb.com at RECS.BSKYB.COM,cn=services,cn=accounts,dc=recs,dc=bskyb,dc=com
> 
> [root at recsds5 sch32]# tail -f /var/log/dirsrv/slapd-RECS-BSKYB-COM/errors
> [23/Jul/2014:12:52:08 +0100] NSMMReplicationPlugin - agmt="cn=meTorecsds4.bskyb.com" (recsds4:389): Consumer failed to replay change (uniqueid 85992d8b-0da911e4-9433f833-313b8581, CSN 53c7ba7e000300100000): Server is unwilling to perform (53). Will retry later.
> [23/Jul/2014:12:52:08 +0100] NSMMReplicationPlugin - agmt="cn=meTorecsds4.bskyb.com" (recsds4:389): Consumer failed to replay change (uniqueid b0838197-0da911e4-9433f833-313b8581, CSN 53c7ba90000000100000): Server is unwilling to perform (53). Will retry later.
> [23/Jul/2014:12:52:16 +0100] NSMMReplicationPlugin - agmt="cn=meTorecsds4.bskyb.com" (recsds4:389): Consumer failed to replay change (uniqueid b0838195-0da911e4-9433f833-313b8581, CSN 53c7ba75000500100000): Server is unwilling to perform (53). Will retry later.
> 
> The background to this is a storage crash caused the master CA IAP to get fudged, and I then proceeded to promote a replica to master CA, re-added crashed IPAs and trying to sync them all up again and clean old orphaned RUVs.
> 
> Regards,
> Suhail Choudhury.
> DevOps | Recommendations Team | BSkyB

Somebody from DS may have a better idea, but it seems to me that the fastest
way to recover is to either "ipa-replica-manage re-initialize" the replicas
from the new CA IPA master (I am assuming this one is running more or less
fine) or even to uninstall, "ipa-replica-manage del" it and install again to
get a clean environment.

Martin