[Freeipa-users] ipa replica failure

[Rob Crittenden]

Andrew E. Bruno wrote:
> On Mon, Jun 22, 2015 at 10:02:59AM -0400, Rob Crittenden wrote:
>> Andrew E. Bruno wrote:
>>> On Fri, Jun 19, 2015 at 03:18:50PM -0400, Rob Crittenden wrote:
>>>> Rich Megginson wrote:
>>>>> On 06/19/2015 12:22 PM, Andrew E. Bruno wrote:
>>>>>>
>>>>>> Questions:
>>>>>>
>>>>>> 0. Is it likely that after running out of file descriptors the dirsrv
>>>>>> slapd database on rep2 was corrupted?
>>>>>
>>>>> That would appear to be the case based on correlation of events,
>>>>> although I've never seen that happen, and it is not supposed to happen.
>>>>>
>>>>>>
>>>>>> 1. Do we have to run ipa-replica-manage del rep2 on *each* of the
>>>>>> remaining replica servers (rep1 and rep3)? Or should it just be run on
>>>>>> the first master?
>>>>>
>>>>> I believe it should only be run on the first master, but it hung, so
>>>>> something is not right, and I'm not sure how to remedy the situation.
>>>>
>>>> How long did it hang, and where?
>>>
>>> This command was run on rep1 (first master):
>>>
>>> [rep1]$ ipa-replica-manage del rep2
>>>
>>> This command hung.. (~10 minutes..) until I Ctr-C. After noticing ldap
>>> queries were hanging on rep2 we ran this on rep2:
>>>
>>> [rep2]$ systemctl stop ipa
>>> (shutdown all ipa services on rep2)
>>>
>>> Then back on rep1 (first master)
>>>
>>> [rep1]$ ipa-replica-manage -v --force del rep2
>>>
>>> Which appeared to work ok.
>>>
>>>>
>>>>>> Do we need to run ipa-csreplicate-manage del as well?
>>>>>>
>>>>>> 2. Why does the rep2 server still appear when querying the
>>>>>> nsDS5ReplicationAgreement in ldap? Is this benign or will this pose
>>>>>> problems
>>>>>> when we go to add rep2 back in?
>>>>>
>>>>> You should remove it.
>>>>
>>>> And ipa-csreplica-manage is the tool to do it.
>>>
>>> When I run this on rep1 (first master):
>>>
>>> [rep1]$ ipa-csreplica-manage list
>>> Directory Manager password:
>>>
>>> rep3: master
>>> rep1: master
>>>
>>>
>>> [rep1]$ ipa-csreplica-manage del rep2
>>> Directory Manager password:
>>>
>>> 'rep1' has no replication agreement for 'rep2'
>>>
>>> But seems to still be there:
>>>
>>> [rep1]$ ldapsearch -Y GSSAPI -b "cn=mapping tree,cn=config" objectClass=nsDS5ReplicationAgreement -LL
>>>
>>> dn: cn=masterAgreement1-rep3-pki-tomcat,cn=replica,cn=ipaca,cn=mapping tree,cn=config
>>> objectClass: top
>>> objectClass: nsds5replicationagreement
>>> cn: masterAgreement1-rep3-pki-tomcat
>>> nsDS5ReplicaRoot: o=ipaca
>>> nsDS5ReplicaHost: rep3
>>> nsDS5ReplicaPort: 389
>>> nsDS5ReplicaBindDN: cn=Replication Manager cloneAgreement1-rep3-pki-tomcat,ou=csusers,cn=config
>>> nsDS5ReplicaBindMethod: Simple
>>> nsDS5ReplicaTransportInfo: TLS
>>> description: masterAgreement1-rep3-pki-tomcat
>>> nsds50ruv: {replicageneration} 5527f74b000000600000
>>> nsds50ruv: {replica 91 ldap://rep3:389} 5537c7ba0000005b
>>>   0000 5582c7e40004005b0000
>>> nsds50ruv: {replica 96 ldap://rep1:389} 5527f75400000060
>>>   0000 5582cd19000000600000
>>> nsds50ruv: {replica 97 ldap://rep2:389} 5527f76000000061
>>>   0000 556f462b000400610000
>>> nsruvReplicaLastModified: {replica 91 ldap://rep3:389} 0
>>>   0000000
>>> nsruvReplicaLastModified: {replica 96 ldap://rep1:389} 0
>>>   0000000
>>> nsruvReplicaLastModified: {replica 97 ldap://rep2:389} 0
>>>   0000000
>>> nsds5replicaLastUpdateStart: 20150619193149Z
>>> nsds5replicaLastUpdateEnd: 20150619193149Z
>>> nsds5replicaChangesSentSinceStartup:: OTY6MTMyLzAg
>>> nsds5replicaLastUpdateStatus: 0 Replica acquired successfully: Incremental upd
>>>   ate succeeded
>>> nsds5replicaUpdateInProgress: FALSE
>>> nsds5replicaLastInitStart: 0
>>> nsds5replicaLastInitEnd: 0
>>>
>>>
>>> However, when I run the ldapsearch on rep3 it's not there (the
>>> cn=ipaca,cn=mapping tree,cn=config is not listed):
>>>
>>> [rep3]$ ldapsearch -Y GSSAPI -b "cn=mapping tree,cn=config" objectClass=nsDS5ReplicationAgreement -LL
>>>
>>> dn: cn=meTorep1,cn=replica,cn=dc\3Dccr\2Cdc\3Dbuffalo\2C dc\3Dedu,cn=mapping tree,cn=config
>>> cn: meTorep1
>>> objectClass: nsds5replicationagreement
>>> objectClass: top
>>> nsDS5ReplicaTransportInfo: LDAP
>>> description: me to rep1
>>> nsDS5ReplicaRoot: dc=ccr,dc=buffalo,dc=edu
>>> nsDS5ReplicaHost: rep1
>>>
>>>
>>>>
>>>>>>
>>>>>> 3. What steps/commands can we take to verify rep2 was successfully
>>>>>> removed and
>>>>>> replication is behaving normally?
>>>>
>>>> The ldapsearch you performed already will confirm that the CA agreement has
>>>> been removed.
>>>
>>> Still showing up.. Any thoughts?
>>>
>>> At this point we want to ensure both remaining masters are functional and
>>> operating normally. Any other commands you recommend running to check?
>>
>> You aren't seeing a replication agreement. You're seeing the Replication
>> Update Vector (RUV).
>>
>> See http://directory.fedoraproject.org/docs/389ds/howto/howto-cleanruv.html
>>
>> You need to do something like:
>>
>> # ldapmodify -D "cn=directory manager" -W -a
>> dn: cn=clean 97, cn=cleanallruv, cn=tasks, cn=config
>> objectclass: extensibleObject
>> replica-base-dn: o=ipaca
>> replica-id: 97
>> cn: clean 97
>>
>
> Great, thanks for the clarification.
>
> Curious what's the difference between running the ldapmodify above and
> ipa-replica-manage clean-ruv?
>

Nothing, for the IPA data. This is a remanant from a CA replication 
agreement and it was an oversight not to add similar RUV management 
options to the ipa-careplica-manage tool.

rob