[Freeipa-users] IPA Replication Status

[Choudhury, Suhail]

Hi Rich,

The version of 389 installed is:

[root at recsds1 sch32]# rpm -q 389-ds-base
389-ds-base-1.2.11.15-33.el6_5.x86_64

Re-initializing didn't work, so I uninstalled and re-installed replicas.

Went through a few rounds of connecting/re-initializing and replication is finally happy.

Also had an issue with GSSAPIAuthentication set to no in SSHD which caused replication errors in the logs as LDAP was explicitly using GSSAPI.

Thanks for your replies all.

Regards,
Suhail Choudhury.
DevOps | Recommendations Team | BSkyB


________________________________________
From: freeipa-users-bounces at redhat.com [freeipa-users-bounces at redhat.com] on behalf of Rich Megginson [rmeggins at redhat.com]
Sent: 23 July 2014 15:16
To: freeipa-users at redhat.com
Subject: Re: [Freeipa-users] IPA Replication Status

On 07/23/2014 06:02 AM, Martin Kosek wrote:
> On 07/23/2014 01:58 PM, Choudhury, Suhail wrote:
>> I have the following errors on different boxes:
>>
>> [root at recsds1 sch32]# tail -f /var/log/dirsrv/slapd-RECS-BSKYB-COM/errors
>> [23/Jul/2014:12:28:54 +0100] NSMMReplicationPlugin - CleanAllRUV Task: Replicas have not been cleaned yet, retrying in 10 seconds
>> [23/Jul/2014:12:29:06 +0100] NSMMReplicationPlugin - CleanAllRUV Task: Waiting for all the replicas to finish cleaning...
>> [23/Jul/2014:12:29:06 +0100] NSMMReplicationPlugin - CleanAllRUV Task: Not all replicas finished cleaning, retrying in 10 seconds
>> [23/Jul/2014:12:29:16 +0100] NSMMReplicationPlugin - CleanAllRUV Task: Not all replicas finished cleaning, retrying in 20 seconds
>> [23/Jul/2014:12:29:36 +0100] NSMMReplicationPlugin - CleanAllRUV Task: Not all replicas finished cleaning, retrying in 40 seconds
>>
>> [root at recsds3 sch32]# tail -f /var/log/dirsrv/slapd-RECS-BSKYB-COM/errors
>> [23/Jul/2014:12:52:10 +0100] agmt="cn=meTorecsds2.bskyb.com" (recsds2:389) - Can't locate CSN 53c7ba27000000100000 in the changelog (DB rc=-30988). The consumer may need to be reinitialized.
>> [23/Jul/2014:12:52:10 +0100] NSMMReplicationPlugin - agmt="cn=meTorecsds2.bskyb.com" (recsds2:389): changelog iteration code returned a dummy entry with csn 53c7c6b1000200100000, skipping ...
>> [23/Jul/2014:12:52:13 +0100] agmt="cn=meTorecsds4.bskyb.com" (recsds4:389) - Can't locate CSN 53c7ba75000400100000 in the changelog (DB rc=-30988). The consumer may need to be reinitialized.
>> [23/Jul/2014:12:52:13 +0100] NSMMReplicationPlugin - agmt="cn=meTorecsds4.bskyb.com" (recsds4:389): changelog iteration code returned a dummy entry with csn 53c7c6b1000200100000, skipping ...
>> [23/Jul/2014:12:52:13 +0100] agmt="cn=meTorecsds2.bskyb.com" (recsds2:389) - Can't locate CSN 53c7ba27000000100000 in the changelog (DB rc=-30988). The consumer may need to be reinitialized.
>>
>> [root at recsds4 ~]# tail -f /var/log/dirsrv/slapd-RECS-BSKYB-COM/errors
>> [23/Jul/2014:12:52:03 +0100] ldbm_back_modify - Attempt to modify a tombstone entry nsuniqueid=b0838195-0da911e4-9433f833-313b8581,krbprincipalname=DNS/recsds1.bskyb.com at RECS.BSKYB.COM,cn=services,cn=accounts,dc=recs,dc=bskyb,dc=com
>> [23/Jul/2014:12:52:03 +0100] ldbm_back_modify - Attempt to modify a tombstone entry nsuniqueid=85992d8b-0da911e4-9433f833-313b8581,fqdn=recsds1.bskyb.com,cn=computers,cn=accounts,dc=recs,dc=bskyb,dc=com
>> [23/Jul/2014:12:52:06 +0100] ldbm_back_modify - Attempt to modify a tombstone entry nsuniqueid=b0838195-0da911e4-9433f833-313b8581,krbprincipalname=DNS/recsds1.bskyb.com at RECS.BSKYB.COM,cn=services,cn=accounts,dc=recs,dc=bskyb,dc=com
>>
>> [root at recsds5 sch32]# tail -f /var/log/dirsrv/slapd-RECS-BSKYB-COM/errors
>> [23/Jul/2014:12:52:08 +0100] NSMMReplicationPlugin - agmt="cn=meTorecsds4.bskyb.com" (recsds4:389): Consumer failed to replay change (uniqueid 85992d8b-0da911e4-9433f833-313b8581, CSN 53c7ba7e000300100000): Server is unwilling to perform (53). Will retry later.
>> [23/Jul/2014:12:52:08 +0100] NSMMReplicationPlugin - agmt="cn=meTorecsds4.bskyb.com" (recsds4:389): Consumer failed to replay change (uniqueid b0838197-0da911e4-9433f833-313b8581, CSN 53c7ba90000000100000): Server is unwilling to perform (53). Will retry later.
>> [23/Jul/2014:12:52:16 +0100] NSMMReplicationPlugin - agmt="cn=meTorecsds4.bskyb.com" (recsds4:389): Consumer failed to replay change (uniqueid b0838195-0da911e4-9433f833-313b8581, CSN 53c7ba75000500100000): Server is unwilling to perform (53). Will retry later.
>>
>> The background to this is a storage crash caused the master CA IAP to get fudged, and I then proceeded to promote a replica to master CA, re-added crashed IPAs and trying to sync them all up again and clean old orphaned RUVs.
>>
>> Regards,
>> Suhail Choudhury.
>> DevOps | Recommendations Team | BSkyB
> Somebody from DS may have a better idea, but it seems to me that the fastest
> way to recover is to either "ipa-replica-manage re-initialize" the replicas
> from the new CA IPA master (I am assuming this one is running more or less
> fine) or even to uninstall, "ipa-replica-manage del" it and install again to
> get a clean environment.

Try the re-initialize first.  That will be necessary since you have the
following error: "The consumer may need to be reinitialized."

Note that "busy" is a normal condition.  A consumer allows updates from
only 1 supplier at a time, and the other suppliers will get a "busy signal".

What version of 389-ds-base are you using?  rpm -q 389-ds-base

>
> Martin
>

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go To http://freeipa.org for more info on the project
Information in this email including any attachments may be privileged, confidential and is intended exclusively for the addressee. The views expressed may not be official policy, but the personal views of the originator. If you have received it in error, please notify the sender by return e-mail and delete it from your system. You should not reproduce, distribute, store, retransmit, use or disclose its contents to anyone. Please note we reserve the right to monitor all e-mail communication through our internal and external networks. SKY and the SKY marks are trademarks of British Sky Broadcasting Group plc and Sky International AG and are used under licence. British Sky Broadcasting Limited (Registration No. 2906991), Sky-In-Home Service Limited (Registration No. 2067075) and Sky Subscribers Services Limited (Registration No. 2340150) are direct or indirect subsidiaries of British Sky Broadcasting Group plc (Registration No. 2247735). All of the companies mentioned in this paragraph are incorporated in England and Wales and share the same registered office at Grant Way, Isleworth, Middlesex TW7 5QD.