[Freeipa-users] IPA Replication Status

[Rich Megginson]

On 07/23/2014 06:02 AM, Martin Kosek wrote:
> On 07/23/2014 01:58 PM, Choudhury, Suhail wrote:
>> I have the following errors on different boxes:
>>
>> [root at recsds1 sch32]# tail -f /var/log/dirsrv/slapd-RECS-BSKYB-COM/errors
>> [23/Jul/2014:12:28:54 +0100] NSMMReplicationPlugin - CleanAllRUV Task: Replicas have not been cleaned yet, retrying in 10 seconds
>> [23/Jul/2014:12:29:06 +0100] NSMMReplicationPlugin - CleanAllRUV Task: Waiting for all the replicas to finish cleaning...
>> [23/Jul/2014:12:29:06 +0100] NSMMReplicationPlugin - CleanAllRUV Task: Not all replicas finished cleaning, retrying in 10 seconds
>> [23/Jul/2014:12:29:16 +0100] NSMMReplicationPlugin - CleanAllRUV Task: Not all replicas finished cleaning, retrying in 20 seconds
>> [23/Jul/2014:12:29:36 +0100] NSMMReplicationPlugin - CleanAllRUV Task: Not all replicas finished cleaning, retrying in 40 seconds
>>
>> [root at recsds3 sch32]# tail -f /var/log/dirsrv/slapd-RECS-BSKYB-COM/errors
>> [23/Jul/2014:12:52:10 +0100] agmt="cn=meTorecsds2.bskyb.com" (recsds2:389) - Can't locate CSN 53c7ba27000000100000 in the changelog (DB rc=-30988). The consumer may need to be reinitialized.
>> [23/Jul/2014:12:52:10 +0100] NSMMReplicationPlugin - agmt="cn=meTorecsds2.bskyb.com" (recsds2:389): changelog iteration code returned a dummy entry with csn 53c7c6b1000200100000, skipping ...
>> [23/Jul/2014:12:52:13 +0100] agmt="cn=meTorecsds4.bskyb.com" (recsds4:389) - Can't locate CSN 53c7ba75000400100000 in the changelog (DB rc=-30988). The consumer may need to be reinitialized.
>> [23/Jul/2014:12:52:13 +0100] NSMMReplicationPlugin - agmt="cn=meTorecsds4.bskyb.com" (recsds4:389): changelog iteration code returned a dummy entry with csn 53c7c6b1000200100000, skipping ...
>> [23/Jul/2014:12:52:13 +0100] agmt="cn=meTorecsds2.bskyb.com" (recsds2:389) - Can't locate CSN 53c7ba27000000100000 in the changelog (DB rc=-30988). The consumer may need to be reinitialized.
>>
>> [root at recsds4 ~]# tail -f /var/log/dirsrv/slapd-RECS-BSKYB-COM/errors
>> [23/Jul/2014:12:52:03 +0100] ldbm_back_modify - Attempt to modify a tombstone entry nsuniqueid=b0838195-0da911e4-9433f833-313b8581,krbprincipalname=DNS/recsds1.bskyb.com at RECS.BSKYB.COM,cn=services,cn=accounts,dc=recs,dc=bskyb,dc=com
>> [23/Jul/2014:12:52:03 +0100] ldbm_back_modify - Attempt to modify a tombstone entry nsuniqueid=85992d8b-0da911e4-9433f833-313b8581,fqdn=recsds1.bskyb.com,cn=computers,cn=accounts,dc=recs,dc=bskyb,dc=com
>> [23/Jul/2014:12:52:06 +0100] ldbm_back_modify - Attempt to modify a tombstone entry nsuniqueid=b0838195-0da911e4-9433f833-313b8581,krbprincipalname=DNS/recsds1.bskyb.com at RECS.BSKYB.COM,cn=services,cn=accounts,dc=recs,dc=bskyb,dc=com
>>
>> [root at recsds5 sch32]# tail -f /var/log/dirsrv/slapd-RECS-BSKYB-COM/errors
>> [23/Jul/2014:12:52:08 +0100] NSMMReplicationPlugin - agmt="cn=meTorecsds4.bskyb.com" (recsds4:389): Consumer failed to replay change (uniqueid 85992d8b-0da911e4-9433f833-313b8581, CSN 53c7ba7e000300100000): Server is unwilling to perform (53). Will retry later.
>> [23/Jul/2014:12:52:08 +0100] NSMMReplicationPlugin - agmt="cn=meTorecsds4.bskyb.com" (recsds4:389): Consumer failed to replay change (uniqueid b0838197-0da911e4-9433f833-313b8581, CSN 53c7ba90000000100000): Server is unwilling to perform (53). Will retry later.
>> [23/Jul/2014:12:52:16 +0100] NSMMReplicationPlugin - agmt="cn=meTorecsds4.bskyb.com" (recsds4:389): Consumer failed to replay change (uniqueid b0838195-0da911e4-9433f833-313b8581, CSN 53c7ba75000500100000): Server is unwilling to perform (53). Will retry later.
>>
>> The background to this is a storage crash caused the master CA IAP to get fudged, and I then proceeded to promote a replica to master CA, re-added crashed IPAs and trying to sync them all up again and clean old orphaned RUVs.
>>
>> Regards,
>> Suhail Choudhury.
>> DevOps | Recommendations Team | BSkyB
> Somebody from DS may have a better idea, but it seems to me that the fastest
> way to recover is to either "ipa-replica-manage re-initialize" the replicas
> from the new CA IPA master (I am assuming this one is running more or less
> fine) or even to uninstall, "ipa-replica-manage del" it and install again to
> get a clean environment.

Try the re-initialize first.  That will be necessary since you have the 
following error: "The consumer may need to be reinitialized."

Note that "busy" is a normal condition.  A consumer allows updates from 
only 1 supplier at a time, and the other suppliers will get a "busy signal".

What version of 389-ds-base are you using?  rpm -q 389-ds-base

>
> Martin
>