[Freeipa-users] ipa replica failure

[Rob Crittenden]

Andrew E. Bruno wrote:
> On Fri, Jun 19, 2015 at 03:18:50PM -0400, Rob Crittenden wrote:
>> Rich Megginson wrote:
>>> On 06/19/2015 12:22 PM, Andrew E. Bruno wrote:
>>>>
>>>> Questions:
>>>>
>>>> 0. Is it likely that after running out of file descriptors the dirsrv
>>>> slapd database on rep2 was corrupted?
>>>
>>> That would appear to be the case based on correlation of events,
>>> although I've never seen that happen, and it is not supposed to happen.
>>>
>>>>
>>>> 1. Do we have to run ipa-replica-manage del rep2 on *each* of the
>>>> remaining replica servers (rep1 and rep3)? Or should it just be run on
>>>> the first master?
>>>
>>> I believe it should only be run on the first master, but it hung, so
>>> something is not right, and I'm not sure how to remedy the situation.
>>
>> How long did it hang, and where?
>
> This command was run on rep1 (first master):
>
> [rep1]$ ipa-replica-manage del rep2
>
> This command hung.. (~10 minutes..) until I Ctr-C. After noticing ldap
> queries were hanging on rep2 we ran this on rep2:
>
> [rep2]$ systemctl stop ipa
> (shutdown all ipa services on rep2)
>
> Then back on rep1 (first master)
>
> [rep1]$ ipa-replica-manage -v --force del rep2
>
> Which appeared to work ok.
>
>>
>>>> Do we need to run ipa-csreplicate-manage del as well?
>>>>
>>>> 2. Why does the rep2 server still appear when querying the
>>>> nsDS5ReplicationAgreement in ldap? Is this benign or will this pose
>>>> problems
>>>> when we go to add rep2 back in?
>>>
>>> You should remove it.
>>
>> And ipa-csreplica-manage is the tool to do it.
>
> When I run this on rep1 (first master):
>
> [rep1]$ ipa-csreplica-manage list
> Directory Manager password:
>
> rep3: master
> rep1: master
>
>
> [rep1]$ ipa-csreplica-manage del rep2
> Directory Manager password:
>
> 'rep1' has no replication agreement for 'rep2'
>
> But seems to still be there:
>
> [rep1]$ ldapsearch -Y GSSAPI -b "cn=mapping tree,cn=config" objectClass=nsDS5ReplicationAgreement -LL
>
> dn: cn=masterAgreement1-rep3-pki-tomcat,cn=replica,cn=ipaca,cn=mapping tree,cn=config
> objectClass: top
> objectClass: nsds5replicationagreement
> cn: masterAgreement1-rep3-pki-tomcat
> nsDS5ReplicaRoot: o=ipaca
> nsDS5ReplicaHost: rep3
> nsDS5ReplicaPort: 389
> nsDS5ReplicaBindDN: cn=Replication Manager cloneAgreement1-rep3-pki-tomcat,ou=csusers,cn=config
> nsDS5ReplicaBindMethod: Simple
> nsDS5ReplicaTransportInfo: TLS
> description: masterAgreement1-rep3-pki-tomcat
> nsds50ruv: {replicageneration} 5527f74b000000600000
> nsds50ruv: {replica 91 ldap://rep3:389} 5537c7ba0000005b
>   0000 5582c7e40004005b0000
> nsds50ruv: {replica 96 ldap://rep1:389} 5527f75400000060
>   0000 5582cd19000000600000
> nsds50ruv: {replica 97 ldap://rep2:389} 5527f76000000061
>   0000 556f462b000400610000
> nsruvReplicaLastModified: {replica 91 ldap://rep3:389} 0
>   0000000
> nsruvReplicaLastModified: {replica 96 ldap://rep1:389} 0
>   0000000
> nsruvReplicaLastModified: {replica 97 ldap://rep2:389} 0
>   0000000
> nsds5replicaLastUpdateStart: 20150619193149Z
> nsds5replicaLastUpdateEnd: 20150619193149Z
> nsds5replicaChangesSentSinceStartup:: OTY6MTMyLzAg
> nsds5replicaLastUpdateStatus: 0 Replica acquired successfully: Incremental upd
>   ate succeeded
> nsds5replicaUpdateInProgress: FALSE
> nsds5replicaLastInitStart: 0
> nsds5replicaLastInitEnd: 0
>
>
> However, when I run the ldapsearch on rep3 it's not there (the
> cn=ipaca,cn=mapping tree,cn=config is not listed):
>
> [rep3]$ ldapsearch -Y GSSAPI -b "cn=mapping tree,cn=config" objectClass=nsDS5ReplicationAgreement -LL
>
> dn: cn=meTorep1,cn=replica,cn=dc\3Dccr\2Cdc\3Dbuffalo\2C dc\3Dedu,cn=mapping tree,cn=config
> cn: meTorep1
> objectClass: nsds5replicationagreement
> objectClass: top
> nsDS5ReplicaTransportInfo: LDAP
> description: me to rep1
> nsDS5ReplicaRoot: dc=ccr,dc=buffalo,dc=edu
> nsDS5ReplicaHost: rep1
>
>
>>
>>>>
>>>> 3. What steps/commands can we take to verify rep2 was successfully
>>>> removed and
>>>> replication is behaving normally?
>>
>> The ldapsearch you performed already will confirm that the CA agreement has
>> been removed.
>
> Still showing up.. Any thoughts?
>
> At this point we want to ensure both remaining masters are functional and
> operating normally. Any other commands you recommend running to check?

You aren't seeing a replication agreement. You're seeing the Replication 
Update Vector (RUV).

See http://directory.fedoraproject.org/docs/389ds/howto/howto-cleanruv.html

You need to do something like:

# ldapmodify -D "cn=directory manager" -W -a
dn: cn=clean 97, cn=cleanallruv, cn=tasks, cn=config
objectclass: extensibleObject
replica-base-dn: o=ipaca
replica-id: 97
cn: clean 97

rob