[Freeipa-users] Replication failing on FreeIPA 4.2.0

Nathan Peters Nathan.Peters at globalrelay.net
Fri Jan 15 07:32:09 UTC 2016 

I think I've finally started to make some progress on this.  I did a lot of googling and found some stuff to run manually in 389 ds through ldapmodify commands to clean RUVs.  During this process the server crashed and when it came back online, suddenly all my ghost RUVs were visible through ipa-replica-manage list-ruv.  It was really strange, I had like 5 of them from winsync agreements that kept failing and needing re-initialization, and another 5 from my earlier re-installations of the 2 other domain controllers.

I ran some more ruv cleanup commands through ldap and they all appear to be gone.  I'm not sure how the crash suddenly made them visible though or why they had to be cleaned through ldapmodify directly and ipa-replica-manage could neither see nor clean them.  Console logs below in case anyone can shed some light on it.  I've re-installed the replicas again, and I'm hoping it doesn't crash in 12 hours like last time ...

--- console output ---

[root at dc1-ipa-dev-van slapd-mydomain-NET]# ipa-replica-manage del dc2-ipa-dev-nvan.mydomain.net --force --cleanup
Connection to 'dc2-ipa-dev-nvan.mydomain.net' failed: Insufficient access: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context Invalid credentials
Forcing removal of dc2-ipa-dev-nvan.mydomain.net
Skipping calculation to determine if one or more masters would be orphaned.
Deleting replication agreements between dc2-ipa-dev-nvan.mydomain.net and dc1-ipa-dev-van.mydomain.net, dc1-ipa-dev-nvan.mydomain.net
Failed to get list of agreements from 'dc2-ipa-dev-nvan.mydomain.net': Insufficient access: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context Invalid credentials
Forcing removal on 'dc1-ipa-dev-van.mydomain.net'
Any DNA range on 'dc2-ipa-dev-nvan.mydomain.net' will be lost
Deleted replication agreement from 'dc1-ipa-dev-van.mydomain.net' to 'dc2-ipa-dev-nvan.mydomain.net'
Failed to determine agreement type for 'dc2-ipa-dev-nvan.mydomain.net': Insufficient access: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context Invalid credentials
There were issues removing a connection for dc2-ipa-dev-nvan.mydomain.net from dc1-ipa-dev-nvan.mydomain.net: local variable 'type1' referenced before assignment
Background task created to clean replication data. This may take a while.
This may be safely interrupted with Ctrl+C
[root at dc1-ipa-dev-van slapd-mydomain-NET]#

[root at dc2-ipa-dev-nvan slapd-mydomain-NET]# ipa-server-install --uninstall

This is a NON REVERSIBLE operation and will delete all data and configuration!

Are you sure you want to continue with the uninstall procedure? [no]: yes

Replication agreements with the following IPA masters found: dc1-ipa-dev-van
.mydomain.net. Removing any replication agreements before uninstalling
the server is strongly recommended. You can remove replication agreements by
running the following command on any other IPA master:
$ ipa-replica-manage del dc2-ipa-dev-nvan.mydomain.net

Are you sure you want to continue with the uninstall procedure? [no]: yes
Shutting down all IPA services
Removing IPA client configuration
Unconfiguring ntpd
Configuring certmonger to stop tracking system certificates for KRA
Configuring certmonger to stop tracking system certificates for CA
Unconfiguring CA
Unconfiguring named
Unconfiguring ipa-dnskeysyncd
Unconfiguring web server
Unconfiguring krb5kdc
Unconfiguring kadmin
Unconfiguring directory server
ipa         : ERROR    Instance removal failed.
ipa         : ERROR    Failed to remove DS instance. You may need to remove instance data manually
Unconfiguring ipa_memcached
Unconfiguring ipa-otpd
[root at dc2-ipa-dev-nvan slapd-mydomain-NET]#

[root at dc1-ipa-dev-van slapd-mydomain-NET]# ipa-csreplica-manage del dc2-ipa-dev-nvan.mydomain.net --force -v
Directory Manager password:

Unable to connect to replica dc2-ipa-dev-nvan.mydomain.net, forcing removal
Failed to get data from 'dc2-ipa-dev-nvan.mydomain.net': cannot connect to 'ldap://dc2-ipa-dev-nvan.mydomain.net:389':
Forcing removal on 'dc1-ipa-dev-van.mydomain.net'
There were issues removing a connection: 'NoneType' object has no attribute 'port'


[root at dc1-ipa-dev-van slapd-mydomain-NET]# ldapsearch -D "cn=directory manager" -W -b "o=ipaca" "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" nscpentrywsi
Enter LDAP Password:
# extended LDIF
#
# LDAPv3
# base <o=ipaca> with scope subtree
# filter: (&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))
# requesting: nscpentrywsi
#

# replica, o\3Dipaca, mapping tree, config
dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
nscpentrywsi: dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
nscpentrywsi: objectClass: top
nscpentrywsi: objectClass: nsDS5Replica
nscpentrywsi: objectClass: extensibleobject
nscpentrywsi: nsDS5ReplicaRoot: o=ipaca
nscpentrywsi: nsDS5ReplicaType: 3
nscpentrywsi: nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-dc1-ipa-dev-nvan.mydomain.net-pki-tomcat,ou=csusers,cn=config
nscpentrywsi: nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-dc2-ipa-dev-nvan.mydomain.net-pki-tomcat,ou=csusers,cn=config
nscpentrywsi: cn: replica
nscpentrywsi: nsDS5ReplicaId: 96
nscpentrywsi: nsDS5Flags: 1
nscpentrywsi: creatorsName: uid=pkidbuser,ou=people,o=ipaca
nscpentrywsi: modifiersName: cn=Multimaster Replication Plugin,cn=plugins,cn=c
 onfig
nscpentrywsi: createTimestamp: 20160114034427Z
nscpentrywsi: modifyTimestamp: 20160115034515Z
nscpentrywsi: nsState:: YAAAAAAAAAA3a5hWAAAAAAAAAAAAAAAAAgAAAAAAAAAAAAAAAAAAAA
 ==
nscpentrywsi: nsDS5ReplicaName: 0c97968e-ba7111e5-b1f1cd78-f19552bb
nscpentrywsi: numSubordinates: 1
nscpentrywsi: nsds50ruv: {replicageneration} 5697199b000000600000
nscpentrywsi: nsds50ruv: {replica 96 ldap://dc1-ipa-dev-van.mydomain.net:389} 569719a0000000600000 56986b35000000600000
nscpentrywsi: nsds50ruv: {replica 76 ldap://dc2-ipa-dev-nvan.mydomain.net:389} 56976b310000004c0000 56976b5c0002004c0000
nscpentrywsi: nsds50ruv: {replica 81 ldap://dc1-ipa-dev-nvan.mydomain.net:389} 5697661a000000510000 56986b55000000510000
nscpentrywsi: nsds50ruv: {replica 86 ldap://dc1-ipa-dev-nvan.mydomain.net:389} 569761d2000000560000 5697620b000500560000
nscpentrywsi: nsds50ruv: {replica 91 ldap://dc2-ipa-dev-nvan.mydomain.net:389} 569738560000005b0000 569738790004005b0000
nscpentrywsi: nsds50ruv: {replica 97 ldap://dc1-ipa-dev-nvan.mydomain.net:389} 569719a4000000610000 569719e6001100610000
nscpentrywsi: nsds5agmtmaxcsn: o=ipaca;masterAgreement1-dc1-ipa-dev-nvan.mydomain.net-pki-tomcat;dc1-ipa-dev-nvan.mydomain.net;389;81;56986b3
 5000000600000
nscpentrywsi: nsruvReplicaLastModified: {replica 96 ldap://dc1-ipa-dev-van.mydomain.net:389} 56986b33
nscpentrywsi: nsruvReplicaLastModified: {replica 76 ldap://dc2-ipa-dev-nvan.mydomain.net:389} 56976b68
nscpentrywsi: nsruvReplicaLastModified: {replica 81 ldap://dc1-ipa-dev-nvan.mydomain.net:389} 56986b54
nscpentrywsi: nsruvReplicaLastModified: {replica 86 ldap://dc1-ipa-dev-nvan.mydomain.net:389} 56976208
nscpentrywsi: nsruvReplicaLastModified: {replica 91 ldap://dc2-ipa-dev-nvan.mydomain.net:389} 56973881
nscpentrywsi: nsruvReplicaLastModified: {replica 97 ldap://dc1-ipa-dev-nvan.mydomain.net:389} 00000000
nscpentrywsi: nsds5ReplicaChangeCount: 1464
nscpentrywsi: nsds5replicareapactive: 0

# search result
search: 2
result: 0 Success

# numResponses: 2
# numEntries: 1
[root at dc1-ipa-dev-van slapd-mydomain-NET]# ipa-replica-manage list-ruv
dc2-ipa-dev-nvan.mydomain.net:389: 10
dc1-ipa-dev-van.mydomain.net:389: 4
dc1-ipa-dev-nvan.mydomain.net:389: 9
[root at dc1-ipa-dev-van slapd-mydomain-NET]# ipa-replica-manage clean-ruv 10
Clean the Replication Update Vector for dc2-ipa-dev-nvan.mydomain.net:389

Cleaning the wrong replica ID will cause that server to no
longer replicate so it may miss updates while the process
is running. It would need to be re-initialized to maintain
consistency. Be very careful.
Continue to clean? [no]: yes
Background task created to clean replication data. This may take a while.
This may be safely interrupted with Ctrl+C
Cleanup task created
[root at dc1-ipa-dev-van slapd-mydomain-NET]# ipa-replica-manage list-ruv
dc1-ipa-dev-van.mydomain.net:389: 4
dc1-ipa-dev-nvan.mydomain.net:389: 9
[root at dc1-ipa-dev-van slapd-mydomain-NET]#

[root at dc1-ipa-dev-van slapd-mydomain-NET]# ldapsearch -D "cn=directory manager" -W -b "o=ipaca" "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" nscpentrywsi
Enter LDAP Password:
# extended LDIF
#
# LDAPv3
# base <o=ipaca> with scope subtree
# filter: (&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))
# requesting: nscpentrywsi
#

# replica, o\3Dipaca, mapping tree, config
dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
nscpentrywsi: dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
nscpentrywsi: objectClass: top
nscpentrywsi: objectClass: nsDS5Replica
nscpentrywsi: objectClass: extensibleobject
nscpentrywsi: nsDS5ReplicaRoot: o=ipaca
nscpentrywsi: nsDS5ReplicaType: 3
nscpentrywsi: nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-dc1-ipa-dev-nvan.mydomain.net-pki-tomcat,ou=csusers,cn=config
nscpentrywsi: nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-dc2-ipa-dev-nvan.mydomain.net-pki-tomcat,ou=csusers,cn=config
nscpentrywsi: cn: replica
nscpentrywsi: nsDS5ReplicaId: 96
nscpentrywsi: nsDS5Flags: 1
nscpentrywsi: creatorsName: uid=pkidbuser,ou=people,o=ipaca
nscpentrywsi: modifiersName: cn=Multimaster Replication Plugin,cn=plugins,cn=c
 onfig
nscpentrywsi: createTimestamp: 20160114034427Z
nscpentrywsi: modifyTimestamp: 20160115034515Z
nscpentrywsi: nsState:: YAAAAAAAAAA3a5hWAAAAAAAAAAAAAAAAAgAAAAAAAAAAAAAAAAAAAA
 ==
nscpentrywsi: nsDS5ReplicaName: 0c97968e-ba7111e5-b1f1cd78-f19552bb
nscpentrywsi: numSubordinates: 1
nscpentrywsi: nsds50ruv: {replicageneration} 5697199b000000600000
nscpentrywsi: nsds50ruv: {replica 96 ldap://dc1-ipa-dev-van.mydomain.net:389} 569719a0000000600000 56986b35000000600000
nscpentrywsi: nsds50ruv: {replica 76 ldap://dc2-ipa-dev-nvan.mydomain.net:389} 56976b310000004c0000 56976b5c0002004c0000
nscpentrywsi: nsds50ruv: {replica 81 ldap://dc1-ipa-dev-nvan.mydomain.net:389} 5697661a000000510000 56986b55000000510000
nscpentrywsi: nsds50ruv: {replica 86 ldap://dc1-ipa-dev-nvan.mydomain.net:389} 569761d2000000560000 5697620b000500560000
nscpentrywsi: nsds50ruv: {replica 91 ldap://dc2-ipa-dev-nvan.mydomain.net:389} 569738560000005b0000 569738790004005b0000
nscpentrywsi: nsds50ruv: {replica 97 ldap://dc1-ipa-dev-nvan.mydomain.net:389} 569719a4000000610000 569719e6001100610000
nscpentrywsi: nsds5agmtmaxcsn: o=ipaca;masterAgreement1-dc1-ipa-dev-nvan.mydomain.net-pki-tomcat;dc1-ipa-dev-nvan.mydomain.net;389;81;56986b3
 5000000600000
nscpentrywsi: nsruvReplicaLastModified: {replica 96 ldap://dc1-ipa-dev-van.mydomain.net:389} 56986b33
nscpentrywsi: nsruvReplicaLastModified: {replica 76 ldap://dc2-ipa-dev-nvan.mydomain.net:389} 56976b68
nscpentrywsi: nsruvReplicaLastModified: {replica 81 ldap://dc1-ipa-dev-nvan.mydomain.net:389} 56986b54
nscpentrywsi: nsruvReplicaLastModified: {replica 86 ldap://dc1-ipa-dev-nvan.mydomain.net:389} 56976208
nscpentrywsi: nsruvReplicaLastModified: {replica 91 ldap://dc2-ipa-dev-nvan.mydomain.net:389} 56973881
nscpentrywsi: nsruvReplicaLastModified: {replica 97 ldap://dc1-ipa-dev-nvan.mydomain.net:389} 00000000
nscpentrywsi: nsds5ReplicaChangeCount: 1464
nscpentrywsi: nsds5replicareapactive: 0

# search result
search: 2
result: 0 Success

# numResponses: 2
# numEntries: 1
[root at dc1-ipa-dev-van slapd-mydomain-NET]#


[root at dc1-ipa-dev-van slapd-mydomain-NET]# ipa-replica-manage del dc1-ipa-dev-nvan.mydomain.net --force --cleanup
Connection to 'dc1-ipa-dev-nvan.mydomain.net' failed: Insufficient access: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context Invalid credentials
Forcing removal of dc1-ipa-dev-nvan.mydomain.net
Skipping calculation to determine if one or more masters would be orphaned.
Deleting replication agreements between dc1-ipa-dev-nvan.mydomain.net and dc1-ipa-dev-van.mydomain.net
Failed to get list of agreements from 'dc1-ipa-dev-nvan.mydomain.net': Insufficient access: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context Invalid credentials
Forcing removal on 'dc1-ipa-dev-van.mydomain.net'
Any DNA range on 'dc1-ipa-dev-nvan.mydomain.net' will be lost
Deleted replication agreement from 'dc1-ipa-dev-van.mydomain.net' to 'dc1-ipa-dev-nvan.mydomain.net'
Background task created to clean replication data. This may take a while.
This may be safely interrupted with Ctrl+C
Failed to cleanup dc1-ipa-dev-nvan.mydomain.net entries: Operations error:
You may need to manually remove them from the tree
[root at dc1-ipa-dev-van slapd-mydomain-NET]#

[root at dc1-ipa-dev-van slapd-mydomain-NET]# ipa-csreplica-manage del dc1-ipa-dev-nvan.mydomain.net --force
Directory Manager password:

Unable to connect to replica dc1-ipa-dev-nvan.mydomain.net, forcing removal
Failed to get data from 'dc1-ipa-dev-nvan.mydomain.net': cannot connect to 'ldap://dc1-ipa-dev-nvan.mydomain.net:389':
Forcing removal on 'dc1-ipa-dev-van.mydomain.net'
There were issues removing a connection: 'NoneType' object has no attribute 'port'

[root at dc1-ipa-dev-nvan slapd-mydomain-NET]# ipa-server-install --uninstall

This is a NON REVERSIBLE operation and will delete all data and configuration!

Are you sure you want to continue with the uninstall procedure? [no]: yes

Replication agreements with the following IPA masters found: dc1-ipa-dev-van
.mydomain.net. Removing any replication agreements before uninstalling
the server is strongly recommended. You can remove replication agreements by
running the following command on any other IPA master:
$ ipa-replica-manage del dc1-ipa-dev-nvan.mydomain.net

Are you sure you want to continue with the uninstall procedure? [no]: yes
Shutting down all IPA services
Removing IPA client configuration
Unconfiguring ntpd
Configuring certmonger to stop tracking system certificates for KRA
Configuring certmonger to stop tracking system certificates for CA
Unconfiguring CA
Unconfiguring named
Unconfiguring ipa-dnskeysyncd
Unconfiguring web server
ipa         : ERROR    Command ''/bin/systemctl' 'restart' 'httpd.service'' returned non-zero exit status 1
[root at dc1-ipa-dev-nvan slapd-mydomain-NET]# ipa-server-install --uninstall

This is a NON REVERSIBLE operation and will delete all data and configuration!

Are you sure you want to continue with the uninstall procedure? [no]: yes

WARNING: Failed to connect to Directory Server to find information about
replication agreements. Uninstallation will continue despite the possible
existing replication agreements.
Shutting down all IPA services
Removing IPA client configuration
Configuring certmonger to stop tracking system certificates for KRA
Configuring certmonger to stop tracking system certificates for CA
Unconfiguring krb5kdc
Unconfiguring kadmin
Unconfiguring directory server
ipa         : ERROR    Instance removal failed.
ipa         : ERROR    Failed to remove DS instance. You may need to remove instance data manually
Unconfiguring ipa_memcached
Unconfiguring ipa-otpd
[root at dc1-ipa-dev-nvan slapd-mydomain-NET]# ipa-server-install --uninstall

This is a NON REVERSIBLE operation and will delete all data and configuration!

Are you sure you want to continue with the uninstall procedure? [no]: yes

WARNING: Failed to connect to Directory Server to find information about
replication agreements. Uninstallation will continue despite the possible
existing replication agreements.
Shutting down all IPA services
Removing IPA client configuration
Configuring certmonger to stop tracking system certificates for KRA
Configuring certmonger to stop tracking system certificates for CA
[root at dc1-ipa-dev-nvan slapd-mydomain-NET]#


[root at dc2-ipa-dev-nvan slapd-mydomain-NET]# ipa-server-install --uninstall

This is a NON REVERSIBLE operation and will delete all data and configuration!

Are you sure you want to continue with the uninstall procedure? [no]: yes

Replication agreements with the following IPA masters found: dc1-ipa-dev-van
.mydomain.net. Removing any replication agreements before uninstalling
the server is strongly recommended. You can remove replication agreements by
running the following command on any other IPA master:
$ ipa-replica-manage del dc2-ipa-dev-nvan.mydomain.net

Are you sure you want to continue with the uninstall procedure? [no]: yes
Shutting down all IPA services
Removing IPA client configuration
Unconfiguring ntpd
Configuring certmonger to stop tracking system certificates for KRA
Configuring certmonger to stop tracking system certificates for CA
Unconfiguring CA
Unconfiguring named
Unconfiguring ipa-dnskeysyncd
Unconfiguring web server
Unconfiguring krb5kdc
Unconfiguring kadmin
Unconfiguring directory server
ipa         : ERROR    Instance removal failed.
ipa         : ERROR    Failed to remove DS instance. You may need to remove instance data manually
Unconfiguring ipa_memcached
Unconfiguring ipa-otpd
[root at dc2-ipa-dev-nvan slapd-mydomain-NET]# ipa-server-install --uninstall

This is a NON REVERSIBLE operation and will delete all data and configuration!

Are you sure you want to continue with the uninstall procedure? [no]: yes

WARNING: Failed to connect to Directory Server to find information about
replication agreements. Uninstallation will continue despite the possible
existing replication agreements.
Shutting down all IPA services
Removing IPA client configuration
Configuring certmonger to stop tracking system certificates for KRA
Configuring certmonger to stop tracking system certificates for CA
[root at dc2-ipa-dev-nvan slapd-mydomain-NET]#


[root at dc1-ipa-dev-van slapd-mydomain-NET]# ipa-replica-manage clean-all-ruv
Usage: ipa-replica-manage [options]

ipa-replica-manage: error: must provide a command [clean-ruv | dnarange-set | list-ruv | dnarange-show | connect | force-sync | list-clean-ruv | disconnect | list | dnanextrange-set | dnanextrange-show | del | re-initialize | abort-clean-ruv]
[root at dc1-ipa-dev-van slapd-mydomain-NET]# ipa-replica-manage clean-ruv 9
Clean the Replication Update Vector for dc1-ipa-dev-nvan.mydomain.net:389

Cleaning the wrong replica ID will cause that server to no
longer replicate so it may miss updates while the process
is running. It would need to be re-initialized to maintain
consistency. Be very careful.
Continue to clean? [no]: yes
Background task created to clean replication data. This may take a while.
This may be safely interrupted with Ctrl+C
Cleanup task created
[root at dc1-ipa-dev-van slapd-mydomain-NET]# ipa-replica-manage list-ruv
unexpected error: Insufficient access: SASL(-14): authorization failure: Invalid credentials
[root at dc1-ipa-dev-van slapd-mydomain-NET]# kdestroy
[root at dc1-ipa-dev-van slapd-mydomain-NET]# kinit nathan.peters
Password for nathan.peters at mydomain.NET:
[root at dc1-ipa-dev-van slapd-mydomain-NET]# ipa-replica-manage list-ruv
dc1-ipa-dev-van.mydomain.net:389: 4
[root at dc1-ipa-dev-van slapd-mydomain-NET]#

[root at dc1-ipa-dev-van slapd-mydomain-NET]# ldapsearch -D "cn=directory manager" -W -b "o=ipaca" "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" nscpentrywsi
Enter LDAP Password:
# extended LDIF
#
# LDAPv3
# base <o=ipaca> with scope subtree
# filter: (&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))
# requesting: nscpentrywsi
#

# replica, o\3Dipaca, mapping tree, config
dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
nscpentrywsi: dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
nscpentrywsi: objectClass: top
nscpentrywsi: objectClass: nsDS5Replica
nscpentrywsi: objectClass: extensibleobject
nscpentrywsi: nsDS5ReplicaRoot: o=ipaca
nscpentrywsi: nsDS5ReplicaType: 3
nscpentrywsi: nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-dc1-ipa-dev-nvan.mydomain.net-pki-tomcat,ou=csusers,cn=config
nscpentrywsi: nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-dc2-ipa-dev-nvan.mydomain.net-pki-tomcat,ou=csusers,cn=config
nscpentrywsi: cn: replica
nscpentrywsi: nsDS5ReplicaId: 96
nscpentrywsi: nsDS5Flags: 1
nscpentrywsi: creatorsName: uid=pkidbuser,ou=people,o=ipaca
nscpentrywsi: modifiersName: cn=Multimaster Replication Plugin,cn=plugins,cn=c
 onfig
nscpentrywsi: createTimestamp: 20160114034427Z
nscpentrywsi: modifyTimestamp: 20160115040015Z
nscpentrywsi: nsState:: YAAAAAAAAAC3bphWAAAAAAAAAAAAAAAAAgAAAAAAAAACAAAAAAAAAA
 ==
nscpentrywsi: nsDS5ReplicaName: 0c97968e-ba7111e5-b1f1cd78-f19552bb
nscpentrywsi: nsds50ruv: {replicageneration} 5697199b000000600000
nscpentrywsi: nsds50ruv: {replica 96 ldap://dc1-ipa-dev-van.mydomain.net:389} 569719a0000000600000 56986eb9000000600000
nscpentrywsi: nsds50ruv: {replica 76 ldap://dc2-ipa-dev-nvan.mydomain.net:389} 56976b310000004c0000 56976b5c0002004c0000
nscpentrywsi: nsds50ruv: {replica 81 ldap://dc1-ipa-dev-nvan.mydomain.net:389} 5697661a000000510000 56986b55000000510000
nscpentrywsi: nsds50ruv: {replica 86 ldap://dc1-ipa-dev-nvan.mydomain.net:389} 569761d2000000560000 5697620b000500560000
nscpentrywsi: nsds50ruv: {replica 91 ldap://dc2-ipa-dev-nvan.mydomain.net:389} 569738560000005b0000 569738790004005b0000
nscpentrywsi: nsds50ruv: {replica 97 ldap://dc1-ipa-dev-nvan.mydomain.net:389} 569719a4000000610000 569719e6001100610000
nscpentrywsi: nsruvReplicaLastModified: {replica 96 ldap://dc1-ipa-dev-van.mydomain.net:389} 56986eb7
nscpentrywsi: nsruvReplicaLastModified: {replica 76 ldap://dc2-ipa-dev-nvan.mydomain.net:389} 56976b68
nscpentrywsi: nsruvReplicaLastModified: {replica 81 ldap://dc1-ipa-dev-nvan.mydomain.net:389} 56986b54
nscpentrywsi: nsruvReplicaLastModified: {replica 86 ldap://dc1-ipa-dev-nvan.mydomain.net:389} 56976208
nscpentrywsi: nsruvReplicaLastModified: {replica 91 ldap://dc2-ipa-dev-nvan.mydomain.net:389} 56973881
nscpentrywsi: nsruvReplicaLastModified: {replica 97 ldap://dc1-ipa-dev-nvan.mydomain.net:389} 00000000
nscpentrywsi: nsds5ReplicaChangeCount: 1465
nscpentrywsi: nsds5replicareapactive: 0

# search result
search: 2
result: 0 Success

# numResponses: 2
# numEntries: 1
[root at dc1-ipa-dev-van slapd-mydomain-NET]#





dn: cn=clean 76, cn=cleanallruv, cn=tasks, cn=config
objectclass: extensibleObject
replica-base-dn: dc=dev-mydomain,dc=net
replica-id: 76
replica-force-cleaning: yes
cn: clean 76


ldapmodify -x -D "cn=directory manager" -W <<EOF
dn: cn=replica,cn=dc\3Ddev-mydomain\2Cdc\3Dnet,cn=mapping tree,cn=config
changetype: modify
replace: nsds5task
nsds5task: CLEANRUV76
EOF

ldapmodify -x -D "cn=directory manager" -W <<EOF
dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
changetype: modify
replace: nsds5task
nsds5task: CLEANRUV76
EOF

ldapmodify -x -D "cn=directory manager" -W <<EOF
dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
changetype: modify
replace: nsds5task
nsds5task: CLEANRUV81
EOF

ldapmodify -x -D "cn=directory manager" -W <<EOF
dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
changetype: modify
replace: nsds5task
nsds5task: CLEANRUV91
EOF

==== SERVER CRASHED HERE ====

[15/Jan/2016:05:21:46 +0000] - acquire_replica, supplier RUV is newer
[15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - windows sync - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): Cancelling linger on the connection
[15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - windows sync - windows_acquire_replica returned success (101)
[15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - windows sync - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): State: ready_to_acquire_replica -> sending_updates
[15/Jan/2016:05:21:46 +0000] - csngen_adjust_time: gen state before 569882e20004:1452835306:0:248
[15/Jan/2016:05:21:46 +0000] - csngen_adjust_time: gen state after 569882e20004:1452835306:0:248
[15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - changelog program - _cl5GetDBFile: found DB object 7ffa5b17b8f0 for database /var/lib/dirsrv/slapd-DEV-mydomain-NET/cldb/e054c085-ede211e4-bf10cd78-f19552bb_553fe9bb000000040000.db
[15/Jan/2016:05:21:46 +0000] - _cl5PositionCursorForReplay (agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389)): Consumer RUV:
[15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): {replicageneration} 553fe9bb000000040000
[15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): {replica 4 ldap://dc1-ipa-dev-van.dev-mydomain.net:389} 553fe9c9000000040000 569882e2000000040000 569881ea
[15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): {replica 3 ldap://dc1-ipa-dev-nvan.dev-mydomain.net:389} 553fe9c4000000030000 5696f872000300030000 00000000
[15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): {replica 5} 56921205000100050000 56972b38000500050000 5698802b
[15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): {replica 6} 56971a3b000000060000 56974fcf000400060000 56988036
[15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): {replica 7} 569738e8000200070000 56975902000100070000 5698803b
[15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): {replica 8} 56976262000000080000 5697639a000000080000 56988049
[15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): {replica 9} 569766ae000000090000 56986c8f000000090000 5698808b
[15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): {replica 10} 56976bc60000000a0000 5698139b0002000a0000 5698807a
[15/Jan/2016:05:21:46 +0000] - _cl5PositionCursorForReplay (agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389)): Supplier RUV:
[15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): {replicageneration} 553fe9bb000000040000
[15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): {replica 4 ldap://dc1-ipa-dev-van.dev-mydomain.net:389} 553fe9c9000000040000 569882e2000200040000 569881ea
[15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): {replica 3} 56846eee000300030000 56846eee000300030000 5698802a
[15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): {replica 5} 56972b38000500050000 56972b38000500050000 5698802a
[15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): {replica 6} 56974fcf000400060000 56974fcf000400060000 5698802a
[15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): {replica 7} 56975902000100070000 56975902000100070000 5698802a
[15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): {replica 8} 5697639a000000080000 5697639a000000080000 5698802a
[15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): {replica 9} 56986c8f000000090000 56986c8f000000090000 5698802a
[15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): {replica 10} 5698139b0002000a0000 5698139b0002000a0000 5698802a
[15/Jan/2016:05:21:46 +0000] agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389) - clcache_get_buffer: found thread private buffer cache 7ffa2c0746a0
[15/Jan/2016:05:21:46 +0000] agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389) - clcache_get_buffer: _pool is 7ffa5b425660 _pool->pl_busy_lists is 7ffa2c075c30 _pool->pl_busy_lists->bl_buffers is 7ffa2c0746a0
[15/Jan/2016:05:21:46 +0000] agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389) - session start: anchorcsn=569882e2000000040000
[15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - changelog program - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): CSN 569882e2000000040000 found, position set for replay
[15/Jan/2016:05:21:46 +0000] agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389) - load=1 rec=1 csn=569882e2000200040000
[15/Jan/2016:05:21:46 +0000] agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389) - clcache_load_buffer: rc=-30988
[15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - windows sync - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): No more updates to send (cl5GetNextOperationToReplay)
[15/Jan/2016:05:21:46 +0000] agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389) - session end: state=5 load=1 sent=1 skipped=0 skipped_new_rid=0 skipped_csn_gt_cons_maxcsn=0 skipped_up_to_date=0 skipped_csn_gt_ruv=0 skipped_csn_covered=0
[15/Jan/2016:05:21:46 +0000] - Calling dirsync search request plugin
[15/Jan/2016:05:21:46 +0000] - Sending dirsync search request
[15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - windows sync - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): Beginning linger on the connection
[15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - windows sync - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): State: sending_updates -> wait_for_changes
[15/Jan/2016:05:21:47 +0000] - _csngen_adjust_local_time: gen state before 569882e20004:1452835306:0:248
[15/Jan/2016:05:21:47 +0000] - _csngen_adjust_local_time: gen state after 569882e30000:1452835307:0:248
[15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - ruv_add_csn_inprogress: successfully inserted csn 569882e3000000040000 into pending list
[15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - Purged state information from entry fqdn=zk1-msg-mbsnap1-nva.dev-mydomain.net,cn=computers,cn=accounts,dc=dev-mydomain,dc=net up to CSN 568f4862000200040000
[15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - changelog program - _cl5GetDBFileByReplicaName: found DB object 7ffa5b17b8f0 for database /var/lib/dirsrv/slapd-DEV-mydomain-NET/cldb/e054c085-ede211e4-bf10cd78-f19552bb_553fe9bb000000040000.db
[15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - changelog program - _cl5GetDBFileByReplicaName: found DB object 7ffa5b17b8f0 for database /var/lib/dirsrv/slapd-DEV-mydomain-NET/cldb/e054c085-ede211e4-bf10cd78-f19552bb_553fe9bb000000040000.db
[15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - ruv_update_ruv: successfully committed csn 569882e3000000040000
[15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - windows sync - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): State: wait_for_changes -> wait_for_changes
[15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - windows sync - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): State: wait_for_changes -> ready_to_acquire_replica
[15/Jan/2016:05:21:47 +0000] - acquire_replica, supplier RUV:
[15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - supplier: {replicageneration} 553fe9bb000000040000
[15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - supplier: {replica 4 ldap://dc1-ipa-dev-van.dev-mydomain.net:389} 553fe9c9000000040000 569882e3000000040000 569881eb
[15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - supplier: {replica 3} 56846eee000300030000 56846eee000300030000 5698802a
[15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - supplier: {replica 5} 56972b38000500050000 56972b38000500050000 5698802a
[15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - supplier: {replica 6} 56974fcf000400060000 56974fcf000400060000 5698802a
[15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - supplier: {replica 7} 56975902000100070000 56975902000100070000 5698802a
[15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - supplier: {replica 8} 5697639a000000080000 5697639a000000080000 5698802a
[15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - supplier: {replica 9} 56986c8f000000090000 56986c8f000000090000 5698802a
[15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - supplier: {replica 10} 5698139b0002000a0000 5698139b0002000a0000 5698802a
[15/Jan/2016:05:21:47 +0000] - acquire_replica, consumer RUV:
[15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - consumer: {replicageneration} 553fe9bb000000040000
[15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - consumer: {replica 4 ldap://dc1-ipa-dev-van.dev-mydomain.net:389} 553fe9c9000000040000 569882e2000200040000 569881ea
[15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - consumer: {replica 3 ldap://dc1-ipa-dev-nvan.dev-mydomain.net:389} 553fe9c4000000030000 5696f872000300030000 00000000
[15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - consumer: {replica 5} 56921205000100050000 56972b38000500050000 5698802b
^C
[root at dc1-ipa-dev-van slapd-DEV-mydomain-NET]# ipa-replica-manage list-ruv
unable to decode: {replica 7} 56975902000100070000 56975902000100070000
unable to decode: {replica 10} 5698139b0002000a0000 5698139b0002000a0000
unable to decode: {replica 5} 56972b38000500050000 56972b38000500050000
unable to decode: {replica 8} 5697639a000000080000 5697639a000000080000
unable to decode: {replica 6} 56974fcf000400060000 56974fcf000400060000
unable to decode: {replica 3} 56846eee000300030000 56846eee000300030000
unable to decode: {replica 9} 56986c8f000000090000 56986c8f000000090000
dc1-ipa-dev-van.dev-mydomain.net:389: 4
[root at dc1-ipa-dev-van slapd-DEV-mydomain-NET]# ipa-replica-manage clean-ruv 7
unable to decode: {replica 7} 56975902000100070000 56975902000100070000
unable to decode: {replica 10} 5698139b0002000a0000 5698139b0002000a0000
unable to decode: {replica 5} 56972b38000500050000 56972b38000500050000
unable to decode: {replica 8} 5697639a000000080000 5697639a000000080000
unable to decode: {replica 6} 56974fcf000400060000 56974fcf000400060000
unable to decode: {replica 3} 56846eee000300030000 56846eee000300030000
unable to decode: {replica 9} 56986c8f000000090000 56986c8f000000090000
Replica ID 7 not found
[root at dc1-ipa-dev-van slapd-DEV-mydomain-NET]# ipa-replica-manage list-ruv
unable to decode: {replica 7} 56975902000100070000 56975902000100070000
unable to decode: {replica 10} 5698139b0002000a0000 5698139b0002000a0000
unable to decode: {replica 5} 56972b38000500050000 56972b38000500050000
unable to decode: {replica 8} 5697639a000000080000 5697639a000000080000
unable to decode: {replica 6} 56974fcf000400060000 56974fcf000400060000
unable to decode: {replica 3} 56846eee000300030000 56846eee000300030000
unable to decode: {replica 9} 56986c8f000000090000 56986c8f000000090000
dc1-ipa-dev-van.dev-mydomain.net:389: 4
[root at dc1-ipa-dev-van slapd-DEV-mydomain-NET]# ldapmodify -D "cn=directory manager" -W -a
Enter LDAP Password:
ldap_bind: Invalid credentials (49)
[root at dc1-ipa-dev-van slapd-DEV-mydomain-NET]# objectclass: extensibleObject
-bash: objectclass:: command not found
[root at dc1-ipa-dev-van slapd-DEV-mydomain-NET]# replica-base-dn: dc=dev-mydomain,dc=net
-bash: replica-base-dn:: command not found
[root at dc1-ipa-dev-van slapd-DEV-mydomain-NET]# replica-id: 7
-bash: replica-id:: command not found
[root at dc1-ipa-dev-van slapd-DEV-mydomain-NET]# cn: clean 7MZKXswIqn3arBMw1xzLl
-bash: cn:: command not found
[root at dc1-ipa-dev-van slapd-DEV-mydomain-NET]# ldapmodify -D "cn=directory manager" -W -a
Enter LDAP Password:
dn: cn=clean 7, cn=cleanallruv, cn=tasks, cn=config
objectclass: extensibleObject
replica-base-dn: dc=dev-mydomain,dc=net
replica-id: 7
cn: clean 7

adding new entry "cn=clean 7, cn=cleanallruv, cn=tasks, cn=config"

[root at dc1-ipa-dev-van slapd-DEV-mydomain-NET]# ipa-replica-manage list-ruv
unable to decode: {replica 5} 56972b38000500050000 56972b38000500050000
unable to decode: {replica 8} 5697639a000000080000 5697639a000000080000
unable to decode: {replica 6} 56974fcf000400060000 56974fcf000400060000
unable to decode: {replica 3} 56846eee000300030000 56846eee000300030000
unable to decode: {replica 9} 56986c8f000000090000 56986c8f000000090000
unable to decode: {replica 10} 5698139b0002000a0000 5698139b0002000a0000
dc1-ipa-dev-van.dev-mydomain.net:389: 4
[root at dc1-ipa-dev-van slapd-DEV-mydomain-NET]# ldapmodify -D "cn=directory manager" -W -a
Enter LDAP Password:
dn: cn=clean 5, cn=cleanallruv, cn=tasks, cn=config
objectclass: extensibleObject
replica-base-dn: dc=dev-mydomain,dc=net
replica-id: 5
cn: clean 5

adding new entry "cn=clean 5, cn=cleanallruv, cn=tasks, cn=config"

dn: cn=clean 8, cn=cleanallruv, cn=tasks, cn=config
objectclass: extensibleObject
replica-base-dn: dc=dev-mydomain,dc=net
replica-id: 8
cn: clean 8

adding new entry "cn=clean 8, cn=cleanallruv, cn=tasks, cn=config"

dn: cn=clean 6, cn=cleanallruv, cn=tasks, cn=config
objectclass: extensibleObject
replica-base-dn: dc=dev-mydomain,dc=net
replica-id: 6
cn: clean 6

adding new entry "cn=clean 6, cn=cleanallruv, cn=tasks, cn=config"

dn: cn=clean 3, cn=cleanallruv, cn=tasks, cn=config
objectclass: extensibleObject
replica-base-dn: dc=dev-mydomain,dc=net
replica-id: 3
cn: clean 3

adding new entry "cn=clean 3, cn=cleanallruv, cn=tasks, cn=config"

dn: cn=clean 9, cn=cleanallruv, cn=tasks, cn=config
objectclass: extensibleObject
replica-base-dn: dc=dev-mydomain,dc=net
replica-id: 9
cn: clean 9

adding new entry "cn=clean 9, cn=cleanallruv, cn=tasks, cn=config"

dn: cn=clean 10, cn=cleanallruv, cn=tasks, cn=config
objectclass: extensibleObject
replica-base-dn: dc=dev-mydomain,dc=net
replica-id: 10
cn: clean 10


[root at dc1-ipa-dev-van slapd-DEV-mydomain-NET]# ldapsearch -D "cn=directory manager" -W -b "o=ipaca" "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" nscpentrywsi
Enter LDAP Password:
# extended LDIF
#
# LDAPv3
# base <o=ipaca> with scope subtree
# filter: (&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))
# requesting: nscpentrywsi
#

# replica, o\3Dipaca, mapping tree, config
dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
nscpentrywsi: dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
nscpentrywsi: objectClass: top
nscpentrywsi: objectClass: nsDS5Replica
nscpentrywsi: objectClass: extensibleobject
nscpentrywsi: nsDS5ReplicaRoot: o=ipaca
nscpentrywsi: nsDS5ReplicaType: 3
nscpentrywsi: nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-dc1-
 ipa-dev-nvan.dev-mydomain.net-pki-tomcat,ou=csusers,cn=config
nscpentrywsi: nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-dc2-
 ipa-dev-nvan.dev-mydomain.net-pki-tomcat,ou=csusers,cn=config
nscpentrywsi: cn: replica
nscpentrywsi: nsDS5ReplicaId: 96
nscpentrywsi: nsDS5Flags: 1
nscpentrywsi: creatorsName: uid=pkidbuser,ou=people,o=ipaca
nscpentrywsi: modifiersName: cn=Multimaster Replication Plugin,cn=plugins,cn=c
 onfig
nscpentrywsi: createTimestamp: 20160114034427Z
nscpentrywsi: modifyTimestamp: 20160115060020Z
nscpentrywsi: nsState:: YAAAAAAAAADXiphWAAAAAAAAAAAAAAAAAgAAAAAAAAABAAAAAAAAAA
 ==
nscpentrywsi: nsDS5ReplicaName: 0c97968e-ba7111e5-b1f1cd78-f19552bb
nscpentrywsi: nsds50ruv: {replicageneration} 5697199b000000600000
nscpentrywsi: nsds50ruv: {replica 96 ldap://dc1-ipa-dev-van.dev-mydomain.ne
 t:389} 569719a0000000600000 56988ad9000000600000
nscpentrywsi: nsds50ruv: {replica 97 ldap://dc1-ipa-dev-nvan.dev-mydomain.n
 et:389} 569719a4000000610000 569719e6001100610000
nscpentrywsi: nsds50ruv: {replica 91} 569738790004005b0000 569738790004005b000
 0
nscpentrywsi: nsds50ruv: {replica 86} 5697620b000500560000 5697620b00050056000
 0
nscpentrywsi: nsruvReplicaLastModified: {replica 96 ldap://dc1-ipa-dev-van.dev
 -mydomain.net:389} 56988ad7
nscpentrywsi: nsruvReplicaLastModified: {replica 97 ldap://dc1-ipa-dev-nvan.de
 v-mydomain.net:389} 00000000
nscpentrywsi: nsruvReplicaLastModified: {replica 91} 5698802a
nscpentrywsi: nsruvReplicaLastModified: {replica 86} 5698802a
nscpentrywsi: nsds5ReplicaChangeCount: 908
nscpentrywsi: nsds5replicareapactive: 0

# search result
search: 2
result: 0 Success

# numResponses: 2
# numEntries: 1
[root at dc1-ipa-dev-van slapd-DEV-mydomain-NET]# ldapmodify -x -D "cn=directory manager" -W <<EOF
dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
changetype: modify
replace: nsds5task
nsds5task: CLEANRUV91
EOF

Enter LDAP Password:
modifying entry "cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config"

[root at dc1-ipa-dev-van slapd-DEV-mydomain-NET]# ldapmodify -x -D "cn=directory manager" -W <<EOF
dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
changetype: modify
replace: nsds5task
nsds5task: CLEANRUV86
EOF

Enter LDAP Password:
modifying entry "cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config"

[root at dc1-ipa-dev-van slapd-DEV-mydomain-NET]# ldapsearch -D "cn=directory manager" -W -b "o=ipaca" "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" nscpentrywsi
Enter LDAP Password:
# extended LDIF
#
# LDAPv3
# base <o=ipaca> with scope subtree
# filter: (&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))
# requesting: nscpentrywsi
#

# replica, o\3Dipaca, mapping tree, config
dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
nscpentrywsi: dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
nscpentrywsi: objectClass: top
nscpentrywsi: objectClass: nsDS5Replica
nscpentrywsi: objectClass: extensibleobject
nscpentrywsi: nsDS5ReplicaRoot: o=ipaca
nscpentrywsi: nsDS5ReplicaType: 3
nscpentrywsi: nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-dc1-
 ipa-dev-nvan.dev-mydomain.net-pki-tomcat,ou=csusers,cn=config
nscpentrywsi: nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-dc2-
 ipa-dev-nvan.dev-mydomain.net-pki-tomcat,ou=csusers,cn=config
nscpentrywsi: cn: replica
nscpentrywsi: nsDS5ReplicaId: 96
nscpentrywsi: nsDS5Flags: 1
nscpentrywsi: creatorsName: uid=pkidbuser,ou=people,o=ipaca
nscpentrywsi: modifiersName: cn=Multimaster Replication Plugin,cn=plugins,cn=c
 onfig
nscpentrywsi: createTimestamp: 20160114034427Z
nscpentrywsi: modifyTimestamp: 20160115061052Z
nscpentrywsi: nsState:: YAAAAAAAAADXiphWAAAAAAAAAAAAAAAAAgAAAAAAAAABAAAAAAAAAA
 ==
nscpentrywsi: nsDS5ReplicaName: 0c97968e-ba7111e5-b1f1cd78-f19552bb
nscpentrywsi: nsds50ruv: {replicageneration} 5697199b000000600000
nscpentrywsi: nsds50ruv: {replica 96 ldap://dc1-ipa-dev-van.dev-mydomain.ne
 t:389} 569719a0000000600000 56988ad9000000600000
nscpentrywsi: nsds50ruv: {replica 97 ldap://dc1-ipa-dev-nvan.dev-mydomain.n
 et:389} 569719a4000000610000 569719e6001100610000
nscpentrywsi: nsruvReplicaLastModified: {replica 96 ldap://dc1-ipa-dev-van.dev
 -mydomain.net:389} 56988ad7
nscpentrywsi: nsruvReplicaLastModified: {replica 97 ldap://dc1-ipa-dev-nvan.de
 v-mydomain.net:389} 00000000
nscpentrywsi: nsds5ReplicaChangeCount: 430
nscpentrywsi: nsds5replicareapactive: 0

# search result
search: 2
result: 0 Success

# numResponses: 2
# numEntries: 1
[root at dc1-ipa-dev-van slapd-DEV-mydomain-NET]#


ldapsearch -xLLL -D "cn=directory manager" -W -b dc=dev-mydomain,dc=net \
 '(&(nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff)(objectclass=nstombstone))'
 
 
ldapmodify -D "cn=directory manager" -W -a
dn: cn=clean 7, cn=cleanallruv, cn=tasks, cn=config
objectclass: extensibleObject
replica-base-dn: dc=dev-mydomain,dc=net
replica-id: 7
cn: clean 7

ldapmodify -D "cn=directory manager" -W -a
dn: cn=clean 5, cn=cleanallruv, cn=tasks, cn=config
objectclass: extensibleObject
replica-base-dn: dc=dev-mydomain,dc=net
replica-id: 5
cn: clean 5

dn: cn=clean 8, cn=cleanallruv, cn=tasks, cn=config
objectclass: extensibleObject
replica-base-dn: dc=dev-mydomain,dc=net
replica-id: 8
cn: clean 8

dn: cn=clean 6, cn=cleanallruv, cn=tasks, cn=config
objectclass: extensibleObject
replica-base-dn: dc=dev-mydomain,dc=net
replica-id: 6
cn: clean 6

dn: cn=clean 3, cn=cleanallruv, cn=tasks, cn=config
objectclass: extensibleObject
replica-base-dn: dc=dev-mydomain,dc=net
replica-id: 3
cn: clean 3

dn: cn=clean 9, cn=cleanallruv, cn=tasks, cn=config
objectclass: extensibleObject
replica-base-dn: dc=dev-mydomain,dc=net
replica-id: 9
cn: clean 9

dn: cn=clean 10, cn=cleanallruv, cn=tasks, cn=config
objectclass: extensibleObject
replica-base-dn: dc=dev-mydomain,dc=net
replica-id: 10
cn: clean 10

dn: cn=clean 86, cn=cleanallruv, cn=tasks, cn=config
objectclass: extensibleObject
replica-base-dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
replica-id: 86
cn: clean 86


-----Original Message-----
From: freeipa-users-bounces at redhat.com [mailto:freeipa-users-bounces at redhat.com] On Behalf Of Nathan Peters
Sent: January-14-16 8:25 PM
To: Rob Crittenden; Ludwig Krispenz; freeipa-users at redhat.com
Subject: Re: [Freeipa-users] Replication failing on FreeIPA 4.2.0

And the saga continues...

In my latest round of trying to fix this, I've attempted to remove the replicas again, this time ensuring to use the --force and --cleanup flags to try to remove the data.  As you can see from the output below, it seems like every possible error that could happen did. Some examples :

Ruvs needed to be manually cleaned.
Ldapsearch reveals that nothing at all has been deleted in the ruv section, and I still have 6 duplicates somehow
ipa         : ERROR    Instance removal failed.
ipa         : ERROR    Failed to remove DS instance. You may need to remove instance data manually
SASL failures while removing or trying to get replication agreements

At this point I think I may need to manually clean all the old data, but I'm not even sure where to start.

Also... When dc1 is alone with no replicas, why does he have a ruv for himself... does he need one ?

And... isn't there supposed to be some kind of clean-all-ruv task or is that not in 4.2.0 but only a later version ?

More information about the Freeipa-users mailing list