[Freeipa-users] ipa replica failure

[Andrew E. Bruno]

On Mon, Jun 22, 2015 at 10:02:59AM -0400, Rob Crittenden wrote:
> Andrew E. Bruno wrote:
> >On Fri, Jun 19, 2015 at 03:18:50PM -0400, Rob Crittenden wrote:
> >>Rich Megginson wrote:
> >>>On 06/19/2015 12:22 PM, Andrew E. Bruno wrote:
> >>>>
> >>>>Questions:
> >>>>
> >>>>0. Is it likely that after running out of file descriptors the dirsrv
> >>>>slapd database on rep2 was corrupted?
> >>>
> >>>That would appear to be the case based on correlation of events,
> >>>although I've never seen that happen, and it is not supposed to happen.
> >>>
> >>>>
> >>>>1. Do we have to run ipa-replica-manage del rep2 on *each* of the
> >>>>remaining replica servers (rep1 and rep3)? Or should it just be run on
> >>>>the first master?
> >>>
> >>>I believe it should only be run on the first master, but it hung, so
> >>>something is not right, and I'm not sure how to remedy the situation.
> >>
> >>How long did it hang, and where?
> >
> >This command was run on rep1 (first master):
> >
> >[rep1]$ ipa-replica-manage del rep2
> >
> >This command hung.. (~10 minutes..) until I Ctr-C. After noticing ldap
> >queries were hanging on rep2 we ran this on rep2:
> >
> >[rep2]$ systemctl stop ipa
> >(shutdown all ipa services on rep2)
> >
> >Then back on rep1 (first master)
> >
> >[rep1]$ ipa-replica-manage -v --force del rep2
> >
> >Which appeared to work ok.
> >
> >>
> >>>>Do we need to run ipa-csreplicate-manage del as well?
> >>>>
> >>>>2. Why does the rep2 server still appear when querying the
> >>>>nsDS5ReplicationAgreement in ldap? Is this benign or will this pose
> >>>>problems
> >>>>when we go to add rep2 back in?
> >>>
> >>>You should remove it.
> >>
> >>And ipa-csreplica-manage is the tool to do it.
> >
> >When I run this on rep1 (first master):
> >
> >[rep1]$ ipa-csreplica-manage list
> >Directory Manager password:
> >
> >rep3: master
> >rep1: master
> >
> >
> >[rep1]$ ipa-csreplica-manage del rep2
> >Directory Manager password:
> >
> >'rep1' has no replication agreement for 'rep2'
> >
> >But seems to still be there:
> >
> >[rep1]$ ldapsearch -Y GSSAPI -b "cn=mapping tree,cn=config" objectClass=nsDS5ReplicationAgreement -LL
> >
> >dn: cn=masterAgreement1-rep3-pki-tomcat,cn=replica,cn=ipaca,cn=mapping tree,cn=config
> >objectClass: top
> >objectClass: nsds5replicationagreement
> >cn: masterAgreement1-rep3-pki-tomcat
> >nsDS5ReplicaRoot: o=ipaca
> >nsDS5ReplicaHost: rep3
> >nsDS5ReplicaPort: 389
> >nsDS5ReplicaBindDN: cn=Replication Manager cloneAgreement1-rep3-pki-tomcat,ou=csusers,cn=config
> >nsDS5ReplicaBindMethod: Simple
> >nsDS5ReplicaTransportInfo: TLS
> >description: masterAgreement1-rep3-pki-tomcat
> >nsds50ruv: {replicageneration} 5527f74b000000600000
> >nsds50ruv: {replica 91 ldap://rep3:389} 5537c7ba0000005b
> >  0000 5582c7e40004005b0000
> >nsds50ruv: {replica 96 ldap://rep1:389} 5527f75400000060
> >  0000 5582cd19000000600000
> >nsds50ruv: {replica 97 ldap://rep2:389} 5527f76000000061
> >  0000 556f462b000400610000
> >nsruvReplicaLastModified: {replica 91 ldap://rep3:389} 0
> >  0000000
> >nsruvReplicaLastModified: {replica 96 ldap://rep1:389} 0
> >  0000000
> >nsruvReplicaLastModified: {replica 97 ldap://rep2:389} 0
> >  0000000
> >nsds5replicaLastUpdateStart: 20150619193149Z
> >nsds5replicaLastUpdateEnd: 20150619193149Z
> >nsds5replicaChangesSentSinceStartup:: OTY6MTMyLzAg
> >nsds5replicaLastUpdateStatus: 0 Replica acquired successfully: Incremental upd
> >  ate succeeded
> >nsds5replicaUpdateInProgress: FALSE
> >nsds5replicaLastInitStart: 0
> >nsds5replicaLastInitEnd: 0
> >
> >
> >However, when I run the ldapsearch on rep3 it's not there (the
> >cn=ipaca,cn=mapping tree,cn=config is not listed):
> >
> >[rep3]$ ldapsearch -Y GSSAPI -b "cn=mapping tree,cn=config" objectClass=nsDS5ReplicationAgreement -LL
> >
> >dn: cn=meTorep1,cn=replica,cn=dc\3Dccr\2Cdc\3Dbuffalo\2C dc\3Dedu,cn=mapping tree,cn=config
> >cn: meTorep1
> >objectClass: nsds5replicationagreement
> >objectClass: top
> >nsDS5ReplicaTransportInfo: LDAP
> >description: me to rep1
> >nsDS5ReplicaRoot: dc=ccr,dc=buffalo,dc=edu
> >nsDS5ReplicaHost: rep1
> >
> >
> >>
> >>>>
> >>>>3. What steps/commands can we take to verify rep2 was successfully
> >>>>removed and
> >>>>replication is behaving normally?
> >>
> >>The ldapsearch you performed already will confirm that the CA agreement has
> >>been removed.
> >
> >Still showing up.. Any thoughts?
> >
> >At this point we want to ensure both remaining masters are functional and
> >operating normally. Any other commands you recommend running to check?
> 
> You aren't seeing a replication agreement. You're seeing the Replication
> Update Vector (RUV).
> 
> See http://directory.fedoraproject.org/docs/389ds/howto/howto-cleanruv.html
> 
> You need to do something like:
> 
> # ldapmodify -D "cn=directory manager" -W -a
> dn: cn=clean 97, cn=cleanallruv, cn=tasks, cn=config
> objectclass: extensibleObject
> replica-base-dn: o=ipaca
> replica-id: 97
> cn: clean 97
> 

Great, thanks for the clarification. 

Curious what's the difference between running the ldapmodify above and
ipa-replica-manage clean-ruv? 



--Andrew