[Freeipa-users] Replication failing on FreeIPA 4.2.0

Ludwig Krispenz lkrispen at redhat.com
Fri Jan 15 08:21:01 UTC 2016 

On 01/15/2016 08:32 AM, Nathan Peters wrote:
> I think I've finally started to make some progress on this.  I did a lot of googling and found some stuff to run manually in 389 ds through ldapmodify commands to clean RUVs.  During this process the server crashed and when it came back online, suddenly all my ghost RUVs were visible through ipa-replica-manage list-ruv.  It was really strange, I had like 5 of them from winsync agreements that kept failing and needing re-initialization, and another 5 from my earlier re-installations of the 2 other domain controllers.
>
> I ran some more ruv cleanup commands through ldap and they all appear to be gone.  I'm not sure how the crash suddenly made them visible though or why they had to be cleaned through ldapmodify directly and ipa-replica-manage could neither see nor clean them.
After a crash the RUV could be rebuilt from the changelog, and the 
changelog could contain references to cleaned ReplicaIds and so they 
came to live again. The cleanallruv task was enhanced to also clean the 
changelog, but this fix is in 1.3.4.2+.
> Console logs below in case anyone can shed some light on it.  I've re-installed the replicas again, and I'm hoping it doesn't crash in 12 hours like last time ...
>
> --- console output ---
>
> [root at dc1-ipa-dev-van slapd-mydomain-NET]# ipa-replica-manage del dc2-ipa-dev-nvan.mydomain.net --force --cleanup
> Connection to 'dc2-ipa-dev-nvan.mydomain.net' failed: Insufficient access: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context Invalid credentials
> Forcing removal of dc2-ipa-dev-nvan.mydomain.net
> Skipping calculation to determine if one or more masters would be orphaned.
> Deleting replication agreements between dc2-ipa-dev-nvan.mydomain.net and dc1-ipa-dev-van.mydomain.net, dc1-ipa-dev-nvan.mydomain.net
> Failed to get list of agreements from 'dc2-ipa-dev-nvan.mydomain.net': Insufficient access: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context Invalid credentials
> Forcing removal on 'dc1-ipa-dev-van.mydomain.net'
> Any DNA range on 'dc2-ipa-dev-nvan.mydomain.net' will be lost
> Deleted replication agreement from 'dc1-ipa-dev-van.mydomain.net' to 'dc2-ipa-dev-nvan.mydomain.net'
> Failed to determine agreement type for 'dc2-ipa-dev-nvan.mydomain.net': Insufficient access: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context Invalid credentials
> There were issues removing a connection for dc2-ipa-dev-nvan.mydomain.net from dc1-ipa-dev-nvan.mydomain.net: local variable 'type1' referenced before assignment
> Background task created to clean replication data. This may take a while.
> This may be safely interrupted with Ctrl+C
> [root at dc1-ipa-dev-van slapd-mydomain-NET]#
>
> [root at dc2-ipa-dev-nvan slapd-mydomain-NET]# ipa-server-install --uninstall
>
> This is a NON REVERSIBLE operation and will delete all data and configuration!
>
> Are you sure you want to continue with the uninstall procedure? [no]: yes
>
> Replication agreements with the following IPA masters found: dc1-ipa-dev-van
> .mydomain.net. Removing any replication agreements before uninstalling
> the server is strongly recommended. You can remove replication agreements by
> running the following command on any other IPA master:
> $ ipa-replica-manage del dc2-ipa-dev-nvan.mydomain.net
>
> Are you sure you want to continue with the uninstall procedure? [no]: yes
> Shutting down all IPA services
> Removing IPA client configuration
> Unconfiguring ntpd
> Configuring certmonger to stop tracking system certificates for KRA
> Configuring certmonger to stop tracking system certificates for CA
> Unconfiguring CA
> Unconfiguring named
> Unconfiguring ipa-dnskeysyncd
> Unconfiguring web server
> Unconfiguring krb5kdc
> Unconfiguring kadmin
> Unconfiguring directory server
> ipa         : ERROR    Instance removal failed.
> ipa         : ERROR    Failed to remove DS instance. You may need to remove instance data manually
> Unconfiguring ipa_memcached
> Unconfiguring ipa-otpd
> [root at dc2-ipa-dev-nvan slapd-mydomain-NET]#
>
> [root at dc1-ipa-dev-van slapd-mydomain-NET]# ipa-csreplica-manage del dc2-ipa-dev-nvan.mydomain.net --force -v
> Directory Manager password:
>
> Unable to connect to replica dc2-ipa-dev-nvan.mydomain.net, forcing removal
> Failed to get data from 'dc2-ipa-dev-nvan.mydomain.net': cannot connect to 'ldap://dc2-ipa-dev-nvan.mydomain.net:389':
> Forcing removal on 'dc1-ipa-dev-van.mydomain.net'
> There were issues removing a connection: 'NoneType' object has no attribute 'port'
>
>
> [root at dc1-ipa-dev-van slapd-mydomain-NET]# ldapsearch -D "cn=directory manager" -W -b "o=ipaca" "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" nscpentrywsi
> Enter LDAP Password:
> # extended LDIF
> #
> # LDAPv3
> # base <o=ipaca> with scope subtree
> # filter: (&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))
> # requesting: nscpentrywsi
> #
>
> # replica, o\3Dipaca, mapping tree, config
> dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
> nscpentrywsi: dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
> nscpentrywsi: objectClass: top
> nscpentrywsi: objectClass: nsDS5Replica
> nscpentrywsi: objectClass: extensibleobject
> nscpentrywsi: nsDS5ReplicaRoot: o=ipaca
> nscpentrywsi: nsDS5ReplicaType: 3
> nscpentrywsi: nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-dc1-ipa-dev-nvan.mydomain.net-pki-tomcat,ou=csusers,cn=config
> nscpentrywsi: nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-dc2-ipa-dev-nvan.mydomain.net-pki-tomcat,ou=csusers,cn=config
> nscpentrywsi: cn: replica
> nscpentrywsi: nsDS5ReplicaId: 96
> nscpentrywsi: nsDS5Flags: 1
> nscpentrywsi: creatorsName: uid=pkidbuser,ou=people,o=ipaca
> nscpentrywsi: modifiersName: cn=Multimaster Replication Plugin,cn=plugins,cn=c
>   onfig
> nscpentrywsi: createTimestamp: 20160114034427Z
> nscpentrywsi: modifyTimestamp: 20160115034515Z
> nscpentrywsi: nsState:: YAAAAAAAAAA3a5hWAAAAAAAAAAAAAAAAAgAAAAAAAAAAAAAAAAAAAA
>   ==
> nscpentrywsi: nsDS5ReplicaName: 0c97968e-ba7111e5-b1f1cd78-f19552bb
> nscpentrywsi: numSubordinates: 1
> nscpentrywsi: nsds50ruv: {replicageneration} 5697199b000000600000
> nscpentrywsi: nsds50ruv: {replica 96 ldap://dc1-ipa-dev-van.mydomain.net:389} 569719a0000000600000 56986b35000000600000
> nscpentrywsi: nsds50ruv: {replica 76 ldap://dc2-ipa-dev-nvan.mydomain.net:389} 56976b310000004c0000 56976b5c0002004c0000
> nscpentrywsi: nsds50ruv: {replica 81 ldap://dc1-ipa-dev-nvan.mydomain.net:389} 5697661a000000510000 56986b55000000510000
> nscpentrywsi: nsds50ruv: {replica 86 ldap://dc1-ipa-dev-nvan.mydomain.net:389} 569761d2000000560000 5697620b000500560000
> nscpentrywsi: nsds50ruv: {replica 91 ldap://dc2-ipa-dev-nvan.mydomain.net:389} 569738560000005b0000 569738790004005b0000
> nscpentrywsi: nsds50ruv: {replica 97 ldap://dc1-ipa-dev-nvan.mydomain.net:389} 569719a4000000610000 569719e6001100610000
> nscpentrywsi: nsds5agmtmaxcsn: o=ipaca;masterAgreement1-dc1-ipa-dev-nvan.mydomain.net-pki-tomcat;dc1-ipa-dev-nvan.mydomain.net;389;81;56986b3
>   5000000600000
> nscpentrywsi: nsruvReplicaLastModified: {replica 96 ldap://dc1-ipa-dev-van.mydomain.net:389} 56986b33
> nscpentrywsi: nsruvReplicaLastModified: {replica 76 ldap://dc2-ipa-dev-nvan.mydomain.net:389} 56976b68
> nscpentrywsi: nsruvReplicaLastModified: {replica 81 ldap://dc1-ipa-dev-nvan.mydomain.net:389} 56986b54
> nscpentrywsi: nsruvReplicaLastModified: {replica 86 ldap://dc1-ipa-dev-nvan.mydomain.net:389} 56976208
> nscpentrywsi: nsruvReplicaLastModified: {replica 91 ldap://dc2-ipa-dev-nvan.mydomain.net:389} 56973881
> nscpentrywsi: nsruvReplicaLastModified: {replica 97 ldap://dc1-ipa-dev-nvan.mydomain.net:389} 00000000
> nscpentrywsi: nsds5ReplicaChangeCount: 1464
> nscpentrywsi: nsds5replicareapactive: 0
>
> # search result
> search: 2
> result: 0 Success
>
> # numResponses: 2
> # numEntries: 1
> [root at dc1-ipa-dev-van slapd-mydomain-NET]# ipa-replica-manage list-ruv
> dc2-ipa-dev-nvan.mydomain.net:389: 10
> dc1-ipa-dev-van.mydomain.net:389: 4
> dc1-ipa-dev-nvan.mydomain.net:389: 9
> [root at dc1-ipa-dev-van slapd-mydomain-NET]# ipa-replica-manage clean-ruv 10
> Clean the Replication Update Vector for dc2-ipa-dev-nvan.mydomain.net:389
>
> Cleaning the wrong replica ID will cause that server to no
> longer replicate so it may miss updates while the process
> is running. It would need to be re-initialized to maintain
> consistency. Be very careful.
> Continue to clean? [no]: yes
> Background task created to clean replication data. This may take a while.
> This may be safely interrupted with Ctrl+C
> Cleanup task created
> [root at dc1-ipa-dev-van slapd-mydomain-NET]# ipa-replica-manage list-ruv
> dc1-ipa-dev-van.mydomain.net:389: 4
> dc1-ipa-dev-nvan.mydomain.net:389: 9
> [root at dc1-ipa-dev-van slapd-mydomain-NET]#
>
> [root at dc1-ipa-dev-van slapd-mydomain-NET]# ldapsearch -D "cn=directory manager" -W -b "o=ipaca" "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" nscpentrywsi
> Enter LDAP Password:
> # extended LDIF
> #
> # LDAPv3
> # base <o=ipaca> with scope subtree
> # filter: (&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))
> # requesting: nscpentrywsi
> #
>
> # replica, o\3Dipaca, mapping tree, config
> dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
> nscpentrywsi: dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
> nscpentrywsi: objectClass: top
> nscpentrywsi: objectClass: nsDS5Replica
> nscpentrywsi: objectClass: extensibleobject
> nscpentrywsi: nsDS5ReplicaRoot: o=ipaca
> nscpentrywsi: nsDS5ReplicaType: 3
> nscpentrywsi: nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-dc1-ipa-dev-nvan.mydomain.net-pki-tomcat,ou=csusers,cn=config
> nscpentrywsi: nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-dc2-ipa-dev-nvan.mydomain.net-pki-tomcat,ou=csusers,cn=config
> nscpentrywsi: cn: replica
> nscpentrywsi: nsDS5ReplicaId: 96
> nscpentrywsi: nsDS5Flags: 1
> nscpentrywsi: creatorsName: uid=pkidbuser,ou=people,o=ipaca
> nscpentrywsi: modifiersName: cn=Multimaster Replication Plugin,cn=plugins,cn=c
>   onfig
> nscpentrywsi: createTimestamp: 20160114034427Z
> nscpentrywsi: modifyTimestamp: 20160115034515Z
> nscpentrywsi: nsState:: YAAAAAAAAAA3a5hWAAAAAAAAAAAAAAAAAgAAAAAAAAAAAAAAAAAAAA
>   ==
> nscpentrywsi: nsDS5ReplicaName: 0c97968e-ba7111e5-b1f1cd78-f19552bb
> nscpentrywsi: numSubordinates: 1
> nscpentrywsi: nsds50ruv: {replicageneration} 5697199b000000600000
> nscpentrywsi: nsds50ruv: {replica 96 ldap://dc1-ipa-dev-van.mydomain.net:389} 569719a0000000600000 56986b35000000600000
> nscpentrywsi: nsds50ruv: {replica 76 ldap://dc2-ipa-dev-nvan.mydomain.net:389} 56976b310000004c0000 56976b5c0002004c0000
> nscpentrywsi: nsds50ruv: {replica 81 ldap://dc1-ipa-dev-nvan.mydomain.net:389} 5697661a000000510000 56986b55000000510000
> nscpentrywsi: nsds50ruv: {replica 86 ldap://dc1-ipa-dev-nvan.mydomain.net:389} 569761d2000000560000 5697620b000500560000
> nscpentrywsi: nsds50ruv: {replica 91 ldap://dc2-ipa-dev-nvan.mydomain.net:389} 569738560000005b0000 569738790004005b0000
> nscpentrywsi: nsds50ruv: {replica 97 ldap://dc1-ipa-dev-nvan.mydomain.net:389} 569719a4000000610000 569719e6001100610000
> nscpentrywsi: nsds5agmtmaxcsn: o=ipaca;masterAgreement1-dc1-ipa-dev-nvan.mydomain.net-pki-tomcat;dc1-ipa-dev-nvan.mydomain.net;389;81;56986b3
>   5000000600000
> nscpentrywsi: nsruvReplicaLastModified: {replica 96 ldap://dc1-ipa-dev-van.mydomain.net:389} 56986b33
> nscpentrywsi: nsruvReplicaLastModified: {replica 76 ldap://dc2-ipa-dev-nvan.mydomain.net:389} 56976b68
> nscpentrywsi: nsruvReplicaLastModified: {replica 81 ldap://dc1-ipa-dev-nvan.mydomain.net:389} 56986b54
> nscpentrywsi: nsruvReplicaLastModified: {replica 86 ldap://dc1-ipa-dev-nvan.mydomain.net:389} 56976208
> nscpentrywsi: nsruvReplicaLastModified: {replica 91 ldap://dc2-ipa-dev-nvan.mydomain.net:389} 56973881
> nscpentrywsi: nsruvReplicaLastModified: {replica 97 ldap://dc1-ipa-dev-nvan.mydomain.net:389} 00000000
> nscpentrywsi: nsds5ReplicaChangeCount: 1464
> nscpentrywsi: nsds5replicareapactive: 0
>
> # search result
> search: 2
> result: 0 Success
>
> # numResponses: 2
> # numEntries: 1
> [root at dc1-ipa-dev-van slapd-mydomain-NET]#
>
>
> [root at dc1-ipa-dev-van slapd-mydomain-NET]# ipa-replica-manage del dc1-ipa-dev-nvan.mydomain.net --force --cleanup
> Connection to 'dc1-ipa-dev-nvan.mydomain.net' failed: Insufficient access: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context Invalid credentials
> Forcing removal of dc1-ipa-dev-nvan.mydomain.net
> Skipping calculation to determine if one or more masters would be orphaned.
> Deleting replication agreements between dc1-ipa-dev-nvan.mydomain.net and dc1-ipa-dev-van.mydomain.net
> Failed to get list of agreements from 'dc1-ipa-dev-nvan.mydomain.net': Insufficient access: SASL(-13): authentication failure: GSSAPI Failure: gss_accept_sec_context Invalid credentials
> Forcing removal on 'dc1-ipa-dev-van.mydomain.net'
> Any DNA range on 'dc1-ipa-dev-nvan.mydomain.net' will be lost
> Deleted replication agreement from 'dc1-ipa-dev-van.mydomain.net' to 'dc1-ipa-dev-nvan.mydomain.net'
> Background task created to clean replication data. This may take a while.
> This may be safely interrupted with Ctrl+C
> Failed to cleanup dc1-ipa-dev-nvan.mydomain.net entries: Operations error:
> You may need to manually remove them from the tree
> [root at dc1-ipa-dev-van slapd-mydomain-NET]#
>
> [root at dc1-ipa-dev-van slapd-mydomain-NET]# ipa-csreplica-manage del dc1-ipa-dev-nvan.mydomain.net --force
> Directory Manager password:
>
> Unable to connect to replica dc1-ipa-dev-nvan.mydomain.net, forcing removal
> Failed to get data from 'dc1-ipa-dev-nvan.mydomain.net': cannot connect to 'ldap://dc1-ipa-dev-nvan.mydomain.net:389':
> Forcing removal on 'dc1-ipa-dev-van.mydomain.net'
> There were issues removing a connection: 'NoneType' object has no attribute 'port'
>
> [root at dc1-ipa-dev-nvan slapd-mydomain-NET]# ipa-server-install --uninstall
>
> This is a NON REVERSIBLE operation and will delete all data and configuration!
>
> Are you sure you want to continue with the uninstall procedure? [no]: yes
>
> Replication agreements with the following IPA masters found: dc1-ipa-dev-van
> .mydomain.net. Removing any replication agreements before uninstalling
> the server is strongly recommended. You can remove replication agreements by
> running the following command on any other IPA master:
> $ ipa-replica-manage del dc1-ipa-dev-nvan.mydomain.net
>
> Are you sure you want to continue with the uninstall procedure? [no]: yes
> Shutting down all IPA services
> Removing IPA client configuration
> Unconfiguring ntpd
> Configuring certmonger to stop tracking system certificates for KRA
> Configuring certmonger to stop tracking system certificates for CA
> Unconfiguring CA
> Unconfiguring named
> Unconfiguring ipa-dnskeysyncd
> Unconfiguring web server
> ipa         : ERROR    Command ''/bin/systemctl' 'restart' 'httpd.service'' returned non-zero exit status 1
> [root at dc1-ipa-dev-nvan slapd-mydomain-NET]# ipa-server-install --uninstall
>
> This is a NON REVERSIBLE operation and will delete all data and configuration!
>
> Are you sure you want to continue with the uninstall procedure? [no]: yes
>
> WARNING: Failed to connect to Directory Server to find information about
> replication agreements. Uninstallation will continue despite the possible
> existing replication agreements.
> Shutting down all IPA services
> Removing IPA client configuration
> Configuring certmonger to stop tracking system certificates for KRA
> Configuring certmonger to stop tracking system certificates for CA
> Unconfiguring krb5kdc
> Unconfiguring kadmin
> Unconfiguring directory server
> ipa         : ERROR    Instance removal failed.
> ipa         : ERROR    Failed to remove DS instance. You may need to remove instance data manually
> Unconfiguring ipa_memcached
> Unconfiguring ipa-otpd
> [root at dc1-ipa-dev-nvan slapd-mydomain-NET]# ipa-server-install --uninstall
>
> This is a NON REVERSIBLE operation and will delete all data and configuration!
>
> Are you sure you want to continue with the uninstall procedure? [no]: yes
>
> WARNING: Failed to connect to Directory Server to find information about
> replication agreements. Uninstallation will continue despite the possible
> existing replication agreements.
> Shutting down all IPA services
> Removing IPA client configuration
> Configuring certmonger to stop tracking system certificates for KRA
> Configuring certmonger to stop tracking system certificates for CA
> [root at dc1-ipa-dev-nvan slapd-mydomain-NET]#
>
>
> [root at dc2-ipa-dev-nvan slapd-mydomain-NET]# ipa-server-install --uninstall
>
> This is a NON REVERSIBLE operation and will delete all data and configuration!
>
> Are you sure you want to continue with the uninstall procedure? [no]: yes
>
> Replication agreements with the following IPA masters found: dc1-ipa-dev-van
> .mydomain.net. Removing any replication agreements before uninstalling
> the server is strongly recommended. You can remove replication agreements by
> running the following command on any other IPA master:
> $ ipa-replica-manage del dc2-ipa-dev-nvan.mydomain.net
>
> Are you sure you want to continue with the uninstall procedure? [no]: yes
> Shutting down all IPA services
> Removing IPA client configuration
> Unconfiguring ntpd
> Configuring certmonger to stop tracking system certificates for KRA
> Configuring certmonger to stop tracking system certificates for CA
> Unconfiguring CA
> Unconfiguring named
> Unconfiguring ipa-dnskeysyncd
> Unconfiguring web server
> Unconfiguring krb5kdc
> Unconfiguring kadmin
> Unconfiguring directory server
> ipa         : ERROR    Instance removal failed.
> ipa         : ERROR    Failed to remove DS instance. You may need to remove instance data manually
> Unconfiguring ipa_memcached
> Unconfiguring ipa-otpd
> [root at dc2-ipa-dev-nvan slapd-mydomain-NET]# ipa-server-install --uninstall
>
> This is a NON REVERSIBLE operation and will delete all data and configuration!
>
> Are you sure you want to continue with the uninstall procedure? [no]: yes
>
> WARNING: Failed to connect to Directory Server to find information about
> replication agreements. Uninstallation will continue despite the possible
> existing replication agreements.
> Shutting down all IPA services
> Removing IPA client configuration
> Configuring certmonger to stop tracking system certificates for KRA
> Configuring certmonger to stop tracking system certificates for CA
> [root at dc2-ipa-dev-nvan slapd-mydomain-NET]#
>
>
> [root at dc1-ipa-dev-van slapd-mydomain-NET]# ipa-replica-manage clean-all-ruv
> Usage: ipa-replica-manage [options]
>
> ipa-replica-manage: error: must provide a command [clean-ruv | dnarange-set | list-ruv | dnarange-show | connect | force-sync | list-clean-ruv | disconnect | list | dnanextrange-set | dnanextrange-show | del | re-initialize | abort-clean-ruv]
> [root at dc1-ipa-dev-van slapd-mydomain-NET]# ipa-replica-manage clean-ruv 9
> Clean the Replication Update Vector for dc1-ipa-dev-nvan.mydomain.net:389
>
> Cleaning the wrong replica ID will cause that server to no
> longer replicate so it may miss updates while the process
> is running. It would need to be re-initialized to maintain
> consistency. Be very careful.
> Continue to clean? [no]: yes
> Background task created to clean replication data. This may take a while.
> This may be safely interrupted with Ctrl+C
> Cleanup task created
> [root at dc1-ipa-dev-van slapd-mydomain-NET]# ipa-replica-manage list-ruv
> unexpected error: Insufficient access: SASL(-14): authorization failure: Invalid credentials
> [root at dc1-ipa-dev-van slapd-mydomain-NET]# kdestroy
> [root at dc1-ipa-dev-van slapd-mydomain-NET]# kinit nathan.peters
> Password for nathan.peters at mydomain.NET:
> [root at dc1-ipa-dev-van slapd-mydomain-NET]# ipa-replica-manage list-ruv
> dc1-ipa-dev-van.mydomain.net:389: 4
> [root at dc1-ipa-dev-van slapd-mydomain-NET]#
>
> [root at dc1-ipa-dev-van slapd-mydomain-NET]# ldapsearch -D "cn=directory manager" -W -b "o=ipaca" "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" nscpentrywsi
> Enter LDAP Password:
> # extended LDIF
> #
> # LDAPv3
> # base <o=ipaca> with scope subtree
> # filter: (&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))
> # requesting: nscpentrywsi
> #
>
> # replica, o\3Dipaca, mapping tree, config
> dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
> nscpentrywsi: dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
> nscpentrywsi: objectClass: top
> nscpentrywsi: objectClass: nsDS5Replica
> nscpentrywsi: objectClass: extensibleobject
> nscpentrywsi: nsDS5ReplicaRoot: o=ipaca
> nscpentrywsi: nsDS5ReplicaType: 3
> nscpentrywsi: nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-dc1-ipa-dev-nvan.mydomain.net-pki-tomcat,ou=csusers,cn=config
> nscpentrywsi: nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-dc2-ipa-dev-nvan.mydomain.net-pki-tomcat,ou=csusers,cn=config
> nscpentrywsi: cn: replica
> nscpentrywsi: nsDS5ReplicaId: 96
> nscpentrywsi: nsDS5Flags: 1
> nscpentrywsi: creatorsName: uid=pkidbuser,ou=people,o=ipaca
> nscpentrywsi: modifiersName: cn=Multimaster Replication Plugin,cn=plugins,cn=c
>   onfig
> nscpentrywsi: createTimestamp: 20160114034427Z
> nscpentrywsi: modifyTimestamp: 20160115040015Z
> nscpentrywsi: nsState:: YAAAAAAAAAC3bphWAAAAAAAAAAAAAAAAAgAAAAAAAAACAAAAAAAAAA
>   ==
> nscpentrywsi: nsDS5ReplicaName: 0c97968e-ba7111e5-b1f1cd78-f19552bb
> nscpentrywsi: nsds50ruv: {replicageneration} 5697199b000000600000
> nscpentrywsi: nsds50ruv: {replica 96 ldap://dc1-ipa-dev-van.mydomain.net:389} 569719a0000000600000 56986eb9000000600000
> nscpentrywsi: nsds50ruv: {replica 76 ldap://dc2-ipa-dev-nvan.mydomain.net:389} 56976b310000004c0000 56976b5c0002004c0000
> nscpentrywsi: nsds50ruv: {replica 81 ldap://dc1-ipa-dev-nvan.mydomain.net:389} 5697661a000000510000 56986b55000000510000
> nscpentrywsi: nsds50ruv: {replica 86 ldap://dc1-ipa-dev-nvan.mydomain.net:389} 569761d2000000560000 5697620b000500560000
> nscpentrywsi: nsds50ruv: {replica 91 ldap://dc2-ipa-dev-nvan.mydomain.net:389} 569738560000005b0000 569738790004005b0000
> nscpentrywsi: nsds50ruv: {replica 97 ldap://dc1-ipa-dev-nvan.mydomain.net:389} 569719a4000000610000 569719e6001100610000
> nscpentrywsi: nsruvReplicaLastModified: {replica 96 ldap://dc1-ipa-dev-van.mydomain.net:389} 56986eb7
> nscpentrywsi: nsruvReplicaLastModified: {replica 76 ldap://dc2-ipa-dev-nvan.mydomain.net:389} 56976b68
> nscpentrywsi: nsruvReplicaLastModified: {replica 81 ldap://dc1-ipa-dev-nvan.mydomain.net:389} 56986b54
> nscpentrywsi: nsruvReplicaLastModified: {replica 86 ldap://dc1-ipa-dev-nvan.mydomain.net:389} 56976208
> nscpentrywsi: nsruvReplicaLastModified: {replica 91 ldap://dc2-ipa-dev-nvan.mydomain.net:389} 56973881
> nscpentrywsi: nsruvReplicaLastModified: {replica 97 ldap://dc1-ipa-dev-nvan.mydomain.net:389} 00000000
> nscpentrywsi: nsds5ReplicaChangeCount: 1465
> nscpentrywsi: nsds5replicareapactive: 0
>
> # search result
> search: 2
> result: 0 Success
>
> # numResponses: 2
> # numEntries: 1
> [root at dc1-ipa-dev-van slapd-mydomain-NET]#
>
>
>
>
>
> dn: cn=clean 76, cn=cleanallruv, cn=tasks, cn=config
> objectclass: extensibleObject
> replica-base-dn: dc=dev-mydomain,dc=net
> replica-id: 76
> replica-force-cleaning: yes
> cn: clean 76
>
>
> ldapmodify -x -D "cn=directory manager" -W <<EOF
> dn: cn=replica,cn=dc\3Ddev-mydomain\2Cdc\3Dnet,cn=mapping tree,cn=config
> changetype: modify
> replace: nsds5task
> nsds5task: CLEANRUV76
> EOF
>
> ldapmodify -x -D "cn=directory manager" -W <<EOF
> dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
> changetype: modify
> replace: nsds5task
> nsds5task: CLEANRUV76
> EOF
>
> ldapmodify -x -D "cn=directory manager" -W <<EOF
> dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
> changetype: modify
> replace: nsds5task
> nsds5task: CLEANRUV81
> EOF
>
> ldapmodify -x -D "cn=directory manager" -W <<EOF
> dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
> changetype: modify
> replace: nsds5task
> nsds5task: CLEANRUV91
> EOF
>
> ==== SERVER CRASHED HERE ====
>
> [15/Jan/2016:05:21:46 +0000] - acquire_replica, supplier RUV is newer
> [15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - windows sync - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): Cancelling linger on the connection
> [15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - windows sync - windows_acquire_replica returned success (101)
> [15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - windows sync - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): State: ready_to_acquire_replica -> sending_updates
> [15/Jan/2016:05:21:46 +0000] - csngen_adjust_time: gen state before 569882e20004:1452835306:0:248
> [15/Jan/2016:05:21:46 +0000] - csngen_adjust_time: gen state after 569882e20004:1452835306:0:248
> [15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - changelog program - _cl5GetDBFile: found DB object 7ffa5b17b8f0 for database /var/lib/dirsrv/slapd-DEV-mydomain-NET/cldb/e054c085-ede211e4-bf10cd78-f19552bb_553fe9bb000000040000.db
> [15/Jan/2016:05:21:46 +0000] - _cl5PositionCursorForReplay (agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389)): Consumer RUV:
> [15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): {replicageneration} 553fe9bb000000040000
> [15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): {replica 4 ldap://dc1-ipa-dev-van.dev-mydomain.net:389} 553fe9c9000000040000 569882e2000000040000 569881ea
> [15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): {replica 3 ldap://dc1-ipa-dev-nvan.dev-mydomain.net:389} 553fe9c4000000030000 5696f872000300030000 00000000
> [15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): {replica 5} 56921205000100050000 56972b38000500050000 5698802b
> [15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): {replica 6} 56971a3b000000060000 56974fcf000400060000 56988036
> [15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): {replica 7} 569738e8000200070000 56975902000100070000 5698803b
> [15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): {replica 8} 56976262000000080000 5697639a000000080000 56988049
> [15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): {replica 9} 569766ae000000090000 56986c8f000000090000 5698808b
> [15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): {replica 10} 56976bc60000000a0000 5698139b0002000a0000 5698807a
> [15/Jan/2016:05:21:46 +0000] - _cl5PositionCursorForReplay (agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389)): Supplier RUV:
> [15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): {replicageneration} 553fe9bb000000040000
> [15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): {replica 4 ldap://dc1-ipa-dev-van.dev-mydomain.net:389} 553fe9c9000000040000 569882e2000200040000 569881ea
> [15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): {replica 3} 56846eee000300030000 56846eee000300030000 5698802a
> [15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): {replica 5} 56972b38000500050000 56972b38000500050000 5698802a
> [15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): {replica 6} 56974fcf000400060000 56974fcf000400060000 5698802a
> [15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): {replica 7} 56975902000100070000 56975902000100070000 5698802a
> [15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): {replica 8} 5697639a000000080000 5697639a000000080000 5698802a
> [15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): {replica 9} 56986c8f000000090000 56986c8f000000090000 5698802a
> [15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): {replica 10} 5698139b0002000a0000 5698139b0002000a0000 5698802a
> [15/Jan/2016:05:21:46 +0000] agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389) - clcache_get_buffer: found thread private buffer cache 7ffa2c0746a0
> [15/Jan/2016:05:21:46 +0000] agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389) - clcache_get_buffer: _pool is 7ffa5b425660 _pool->pl_busy_lists is 7ffa2c075c30 _pool->pl_busy_lists->bl_buffers is 7ffa2c0746a0
> [15/Jan/2016:05:21:46 +0000] agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389) - session start: anchorcsn=569882e2000000040000
> [15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - changelog program - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): CSN 569882e2000000040000 found, position set for replay
> [15/Jan/2016:05:21:46 +0000] agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389) - load=1 rec=1 csn=569882e2000200040000
> [15/Jan/2016:05:21:46 +0000] agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389) - clcache_load_buffer: rc=-30988
> [15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - windows sync - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): No more updates to send (cl5GetNextOperationToReplay)
> [15/Jan/2016:05:21:46 +0000] agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389) - session end: state=5 load=1 sent=1 skipped=0 skipped_new_rid=0 skipped_csn_gt_cons_maxcsn=0 skipped_up_to_date=0 skipped_csn_gt_ruv=0 skipped_csn_covered=0
> [15/Jan/2016:05:21:46 +0000] - Calling dirsync search request plugin
> [15/Jan/2016:05:21:46 +0000] - Sending dirsync search request
> [15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - windows sync - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): Beginning linger on the connection
> [15/Jan/2016:05:21:46 +0000] NSMMReplicationPlugin - windows sync - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): State: sending_updates -> wait_for_changes
> [15/Jan/2016:05:21:47 +0000] - _csngen_adjust_local_time: gen state before 569882e20004:1452835306:0:248
> [15/Jan/2016:05:21:47 +0000] - _csngen_adjust_local_time: gen state after 569882e30000:1452835307:0:248
> [15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - ruv_add_csn_inprogress: successfully inserted csn 569882e3000000040000 into pending list
> [15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - Purged state information from entry fqdn=zk1-msg-mbsnap1-nva.dev-mydomain.net,cn=computers,cn=accounts,dc=dev-mydomain,dc=net up to CSN 568f4862000200040000
> [15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - changelog program - _cl5GetDBFileByReplicaName: found DB object 7ffa5b17b8f0 for database /var/lib/dirsrv/slapd-DEV-mydomain-NET/cldb/e054c085-ede211e4-bf10cd78-f19552bb_553fe9bb000000040000.db
> [15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - changelog program - _cl5GetDBFileByReplicaName: found DB object 7ffa5b17b8f0 for database /var/lib/dirsrv/slapd-DEV-mydomain-NET/cldb/e054c085-ede211e4-bf10cd78-f19552bb_553fe9bb000000040000.db
> [15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - ruv_update_ruv: successfully committed csn 569882e3000000040000
> [15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - windows sync - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): State: wait_for_changes -> wait_for_changes
> [15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - windows sync - agmt="cn=meToofficedc2.office.mydomain.net" (officedc2:389): State: wait_for_changes -> ready_to_acquire_replica
> [15/Jan/2016:05:21:47 +0000] - acquire_replica, supplier RUV:
> [15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - supplier: {replicageneration} 553fe9bb000000040000
> [15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - supplier: {replica 4 ldap://dc1-ipa-dev-van.dev-mydomain.net:389} 553fe9c9000000040000 569882e3000000040000 569881eb
> [15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - supplier: {replica 3} 56846eee000300030000 56846eee000300030000 5698802a
> [15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - supplier: {replica 5} 56972b38000500050000 56972b38000500050000 5698802a
> [15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - supplier: {replica 6} 56974fcf000400060000 56974fcf000400060000 5698802a
> [15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - supplier: {replica 7} 56975902000100070000 56975902000100070000 5698802a
> [15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - supplier: {replica 8} 5697639a000000080000 5697639a000000080000 5698802a
> [15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - supplier: {replica 9} 56986c8f000000090000 56986c8f000000090000 5698802a
> [15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - supplier: {replica 10} 5698139b0002000a0000 5698139b0002000a0000 5698802a
> [15/Jan/2016:05:21:47 +0000] - acquire_replica, consumer RUV:
> [15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - consumer: {replicageneration} 553fe9bb000000040000
> [15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - consumer: {replica 4 ldap://dc1-ipa-dev-van.dev-mydomain.net:389} 553fe9c9000000040000 569882e2000200040000 569881ea
> [15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - consumer: {replica 3 ldap://dc1-ipa-dev-nvan.dev-mydomain.net:389} 553fe9c4000000030000 5696f872000300030000 00000000
> [15/Jan/2016:05:21:47 +0000] NSMMReplicationPlugin - consumer: {replica 5} 56921205000100050000 56972b38000500050000 5698802b
> ^C
> [root at dc1-ipa-dev-van slapd-DEV-mydomain-NET]# ipa-replica-manage list-ruv
> unable to decode: {replica 7} 56975902000100070000 56975902000100070000
> unable to decode: {replica 10} 5698139b0002000a0000 5698139b0002000a0000
> unable to decode: {replica 5} 56972b38000500050000 56972b38000500050000
> unable to decode: {replica 8} 5697639a000000080000 5697639a000000080000
> unable to decode: {replica 6} 56974fcf000400060000 56974fcf000400060000
> unable to decode: {replica 3} 56846eee000300030000 56846eee000300030000
> unable to decode: {replica 9} 56986c8f000000090000 56986c8f000000090000
> dc1-ipa-dev-van.dev-mydomain.net:389: 4
> [root at dc1-ipa-dev-van slapd-DEV-mydomain-NET]# ipa-replica-manage clean-ruv 7
> unable to decode: {replica 7} 56975902000100070000 56975902000100070000
> unable to decode: {replica 10} 5698139b0002000a0000 5698139b0002000a0000
> unable to decode: {replica 5} 56972b38000500050000 56972b38000500050000
> unable to decode: {replica 8} 5697639a000000080000 5697639a000000080000
> unable to decode: {replica 6} 56974fcf000400060000 56974fcf000400060000
> unable to decode: {replica 3} 56846eee000300030000 56846eee000300030000
> unable to decode: {replica 9} 56986c8f000000090000 56986c8f000000090000
> Replica ID 7 not found
> [root at dc1-ipa-dev-van slapd-DEV-mydomain-NET]# ipa-replica-manage list-ruv
> unable to decode: {replica 7} 56975902000100070000 56975902000100070000
> unable to decode: {replica 10} 5698139b0002000a0000 5698139b0002000a0000
> unable to decode: {replica 5} 56972b38000500050000 56972b38000500050000
> unable to decode: {replica 8} 5697639a000000080000 5697639a000000080000
> unable to decode: {replica 6} 56974fcf000400060000 56974fcf000400060000
> unable to decode: {replica 3} 56846eee000300030000 56846eee000300030000
> unable to decode: {replica 9} 56986c8f000000090000 56986c8f000000090000
> dc1-ipa-dev-van.dev-mydomain.net:389: 4
> [root at dc1-ipa-dev-van slapd-DEV-mydomain-NET]# ldapmodify -D "cn=directory manager" -W -a
> Enter LDAP Password:
> ldap_bind: Invalid credentials (49)
> [root at dc1-ipa-dev-van slapd-DEV-mydomain-NET]# objectclass: extensibleObject
> -bash: objectclass:: command not found
> [root at dc1-ipa-dev-van slapd-DEV-mydomain-NET]# replica-base-dn: dc=dev-mydomain,dc=net
> -bash: replica-base-dn:: command not found
> [root at dc1-ipa-dev-van slapd-DEV-mydomain-NET]# replica-id: 7
> -bash: replica-id:: command not found
> [root at dc1-ipa-dev-van slapd-DEV-mydomain-NET]# cn: clean 7MZKXswIqn3arBMw1xzLl
> -bash: cn:: command not found
> [root at dc1-ipa-dev-van slapd-DEV-mydomain-NET]# ldapmodify -D "cn=directory manager" -W -a
> Enter LDAP Password:
> dn: cn=clean 7, cn=cleanallruv, cn=tasks, cn=config
> objectclass: extensibleObject
> replica-base-dn: dc=dev-mydomain,dc=net
> replica-id: 7
> cn: clean 7
>
> adding new entry "cn=clean 7, cn=cleanallruv, cn=tasks, cn=config"
>
> [root at dc1-ipa-dev-van slapd-DEV-mydomain-NET]# ipa-replica-manage list-ruv
> unable to decode: {replica 5} 56972b38000500050000 56972b38000500050000
> unable to decode: {replica 8} 5697639a000000080000 5697639a000000080000
> unable to decode: {replica 6} 56974fcf000400060000 56974fcf000400060000
> unable to decode: {replica 3} 56846eee000300030000 56846eee000300030000
> unable to decode: {replica 9} 56986c8f000000090000 56986c8f000000090000
> unable to decode: {replica 10} 5698139b0002000a0000 5698139b0002000a0000
> dc1-ipa-dev-van.dev-mydomain.net:389: 4
> [root at dc1-ipa-dev-van slapd-DEV-mydomain-NET]# ldapmodify -D "cn=directory manager" -W -a
> Enter LDAP Password:
> dn: cn=clean 5, cn=cleanallruv, cn=tasks, cn=config
> objectclass: extensibleObject
> replica-base-dn: dc=dev-mydomain,dc=net
> replica-id: 5
> cn: clean 5
>
> adding new entry "cn=clean 5, cn=cleanallruv, cn=tasks, cn=config"
>
> dn: cn=clean 8, cn=cleanallruv, cn=tasks, cn=config
> objectclass: extensibleObject
> replica-base-dn: dc=dev-mydomain,dc=net
> replica-id: 8
> cn: clean 8
>
> adding new entry "cn=clean 8, cn=cleanallruv, cn=tasks, cn=config"
>
> dn: cn=clean 6, cn=cleanallruv, cn=tasks, cn=config
> objectclass: extensibleObject
> replica-base-dn: dc=dev-mydomain,dc=net
> replica-id: 6
> cn: clean 6
>
> adding new entry "cn=clean 6, cn=cleanallruv, cn=tasks, cn=config"
>
> dn: cn=clean 3, cn=cleanallruv, cn=tasks, cn=config
> objectclass: extensibleObject
> replica-base-dn: dc=dev-mydomain,dc=net
> replica-id: 3
> cn: clean 3
>
> adding new entry "cn=clean 3, cn=cleanallruv, cn=tasks, cn=config"
>
> dn: cn=clean 9, cn=cleanallruv, cn=tasks, cn=config
> objectclass: extensibleObject
> replica-base-dn: dc=dev-mydomain,dc=net
> replica-id: 9
> cn: clean 9
>
> adding new entry "cn=clean 9, cn=cleanallruv, cn=tasks, cn=config"
>
> dn: cn=clean 10, cn=cleanallruv, cn=tasks, cn=config
> objectclass: extensibleObject
> replica-base-dn: dc=dev-mydomain,dc=net
> replica-id: 10
> cn: clean 10
>
>
> [root at dc1-ipa-dev-van slapd-DEV-mydomain-NET]# ldapsearch -D "cn=directory manager" -W -b "o=ipaca" "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" nscpentrywsi
> Enter LDAP Password:
> # extended LDIF
> #
> # LDAPv3
> # base <o=ipaca> with scope subtree
> # filter: (&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))
> # requesting: nscpentrywsi
> #
>
> # replica, o\3Dipaca, mapping tree, config
> dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
> nscpentrywsi: dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
> nscpentrywsi: objectClass: top
> nscpentrywsi: objectClass: nsDS5Replica
> nscpentrywsi: objectClass: extensibleobject
> nscpentrywsi: nsDS5ReplicaRoot: o=ipaca
> nscpentrywsi: nsDS5ReplicaType: 3
> nscpentrywsi: nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-dc1-
>   ipa-dev-nvan.dev-mydomain.net-pki-tomcat,ou=csusers,cn=config
> nscpentrywsi: nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-dc2-
>   ipa-dev-nvan.dev-mydomain.net-pki-tomcat,ou=csusers,cn=config
> nscpentrywsi: cn: replica
> nscpentrywsi: nsDS5ReplicaId: 96
> nscpentrywsi: nsDS5Flags: 1
> nscpentrywsi: creatorsName: uid=pkidbuser,ou=people,o=ipaca
> nscpentrywsi: modifiersName: cn=Multimaster Replication Plugin,cn=plugins,cn=c
>   onfig
> nscpentrywsi: createTimestamp: 20160114034427Z
> nscpentrywsi: modifyTimestamp: 20160115060020Z
> nscpentrywsi: nsState:: YAAAAAAAAADXiphWAAAAAAAAAAAAAAAAAgAAAAAAAAABAAAAAAAAAA
>   ==
> nscpentrywsi: nsDS5ReplicaName: 0c97968e-ba7111e5-b1f1cd78-f19552bb
> nscpentrywsi: nsds50ruv: {replicageneration} 5697199b000000600000
> nscpentrywsi: nsds50ruv: {replica 96 ldap://dc1-ipa-dev-van.dev-mydomain.ne
>   t:389} 569719a0000000600000 56988ad9000000600000
> nscpentrywsi: nsds50ruv: {replica 97 ldap://dc1-ipa-dev-nvan.dev-mydomain.n
>   et:389} 569719a4000000610000 569719e6001100610000
> nscpentrywsi: nsds50ruv: {replica 91} 569738790004005b0000 569738790004005b000
>   0
> nscpentrywsi: nsds50ruv: {replica 86} 5697620b000500560000 5697620b00050056000
>   0
> nscpentrywsi: nsruvReplicaLastModified: {replica 96 ldap://dc1-ipa-dev-van.dev
>   -mydomain.net:389} 56988ad7
> nscpentrywsi: nsruvReplicaLastModified: {replica 97 ldap://dc1-ipa-dev-nvan.de
>   v-mydomain.net:389} 00000000
> nscpentrywsi: nsruvReplicaLastModified: {replica 91} 5698802a
> nscpentrywsi: nsruvReplicaLastModified: {replica 86} 5698802a
> nscpentrywsi: nsds5ReplicaChangeCount: 908
> nscpentrywsi: nsds5replicareapactive: 0
>
> # search result
> search: 2
> result: 0 Success
>
> # numResponses: 2
> # numEntries: 1
> [root at dc1-ipa-dev-van slapd-DEV-mydomain-NET]# ldapmodify -x -D "cn=directory manager" -W <<EOF
> dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
> changetype: modify
> replace: nsds5task
> nsds5task: CLEANRUV91
> EOF
>
> Enter LDAP Password:
> modifying entry "cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config"
>
> [root at dc1-ipa-dev-van slapd-DEV-mydomain-NET]# ldapmodify -x -D "cn=directory manager" -W <<EOF
> dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
> changetype: modify
> replace: nsds5task
> nsds5task: CLEANRUV86
> EOF
>
> Enter LDAP Password:
> modifying entry "cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config"
>
> [root at dc1-ipa-dev-van slapd-DEV-mydomain-NET]# ldapsearch -D "cn=directory manager" -W -b "o=ipaca" "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))" nscpentrywsi
> Enter LDAP Password:
> # extended LDIF
> #
> # LDAPv3
> # base <o=ipaca> with scope subtree
> # filter: (&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))
> # requesting: nscpentrywsi
> #
>
> # replica, o\3Dipaca, mapping tree, config
> dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
> nscpentrywsi: dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
> nscpentrywsi: objectClass: top
> nscpentrywsi: objectClass: nsDS5Replica
> nscpentrywsi: objectClass: extensibleobject
> nscpentrywsi: nsDS5ReplicaRoot: o=ipaca
> nscpentrywsi: nsDS5ReplicaType: 3
> nscpentrywsi: nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-dc1-
>   ipa-dev-nvan.dev-mydomain.net-pki-tomcat,ou=csusers,cn=config
> nscpentrywsi: nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-dc2-
>   ipa-dev-nvan.dev-mydomain.net-pki-tomcat,ou=csusers,cn=config
> nscpentrywsi: cn: replica
> nscpentrywsi: nsDS5ReplicaId: 96
> nscpentrywsi: nsDS5Flags: 1
> nscpentrywsi: creatorsName: uid=pkidbuser,ou=people,o=ipaca
> nscpentrywsi: modifiersName: cn=Multimaster Replication Plugin,cn=plugins,cn=c
>   onfig
> nscpentrywsi: createTimestamp: 20160114034427Z
> nscpentrywsi: modifyTimestamp: 20160115061052Z
> nscpentrywsi: nsState:: YAAAAAAAAADXiphWAAAAAAAAAAAAAAAAAgAAAAAAAAABAAAAAAAAAA
>   ==
> nscpentrywsi: nsDS5ReplicaName: 0c97968e-ba7111e5-b1f1cd78-f19552bb
> nscpentrywsi: nsds50ruv: {replicageneration} 5697199b000000600000
> nscpentrywsi: nsds50ruv: {replica 96 ldap://dc1-ipa-dev-van.dev-mydomain.ne
>   t:389} 569719a0000000600000 56988ad9000000600000
> nscpentrywsi: nsds50ruv: {replica 97 ldap://dc1-ipa-dev-nvan.dev-mydomain.n
>   et:389} 569719a4000000610000 569719e6001100610000
> nscpentrywsi: nsruvReplicaLastModified: {replica 96 ldap://dc1-ipa-dev-van.dev
>   -mydomain.net:389} 56988ad7
> nscpentrywsi: nsruvReplicaLastModified: {replica 97 ldap://dc1-ipa-dev-nvan.de
>   v-mydomain.net:389} 00000000
> nscpentrywsi: nsds5ReplicaChangeCount: 430
> nscpentrywsi: nsds5replicareapactive: 0
>
> # search result
> search: 2
> result: 0 Success
>
> # numResponses: 2
> # numEntries: 1
> [root at dc1-ipa-dev-van slapd-DEV-mydomain-NET]#
>
>
> ldapsearch -xLLL -D "cn=directory manager" -W -b dc=dev-mydomain,dc=net \
>   '(&(nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff)(objectclass=nstombstone))'
>   
>   
> ldapmodify -D "cn=directory manager" -W -a
> dn: cn=clean 7, cn=cleanallruv, cn=tasks, cn=config
> objectclass: extensibleObject
> replica-base-dn: dc=dev-mydomain,dc=net
> replica-id: 7
> cn: clean 7
>
> ldapmodify -D "cn=directory manager" -W -a
> dn: cn=clean 5, cn=cleanallruv, cn=tasks, cn=config
> objectclass: extensibleObject
> replica-base-dn: dc=dev-mydomain,dc=net
> replica-id: 5
> cn: clean 5
>
> dn: cn=clean 8, cn=cleanallruv, cn=tasks, cn=config
> objectclass: extensibleObject
> replica-base-dn: dc=dev-mydomain,dc=net
> replica-id: 8
> cn: clean 8
>
> dn: cn=clean 6, cn=cleanallruv, cn=tasks, cn=config
> objectclass: extensibleObject
> replica-base-dn: dc=dev-mydomain,dc=net
> replica-id: 6
> cn: clean 6
>
> dn: cn=clean 3, cn=cleanallruv, cn=tasks, cn=config
> objectclass: extensibleObject
> replica-base-dn: dc=dev-mydomain,dc=net
> replica-id: 3
> cn: clean 3
>
> dn: cn=clean 9, cn=cleanallruv, cn=tasks, cn=config
> objectclass: extensibleObject
> replica-base-dn: dc=dev-mydomain,dc=net
> replica-id: 9
> cn: clean 9
>
> dn: cn=clean 10, cn=cleanallruv, cn=tasks, cn=config
> objectclass: extensibleObject
> replica-base-dn: dc=dev-mydomain,dc=net
> replica-id: 10
> cn: clean 10
>
> dn: cn=clean 86, cn=cleanallruv, cn=tasks, cn=config
> objectclass: extensibleObject
> replica-base-dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
> replica-id: 86
> cn: clean 86
>
>
> -----Original Message-----
> From: freeipa-users-bounces at redhat.com [mailto:freeipa-users-bounces at redhat.com] On Behalf Of Nathan Peters
> Sent: January-14-16 8:25 PM
> To: Rob Crittenden; Ludwig Krispenz; freeipa-users at redhat.com
> Subject: Re: [Freeipa-users] Replication failing on FreeIPA 4.2.0
>
> And the saga continues...
>
> In my latest round of trying to fix this, I've attempted to remove the replicas again, this time ensuring to use the --force and --cleanup flags to try to remove the data.  As you can see from the output below, it seems like every possible error that could happen did. Some examples :
>
> Ruvs needed to be manually cleaned.
> Ldapsearch reveals that nothing at all has been deleted in the ruv section, and I still have 6 duplicates somehow
> ipa         : ERROR    Instance removal failed.
> ipa         : ERROR    Failed to remove DS instance. You may need to remove instance data manually
> SASL failures while removing or trying to get replication agreements
>
> At this point I think I may need to manually clean all the old data, but I'm not even sure where to start.
>
> Also... When dc1 is alone with no replicas, why does he have a ruv for himself... does he need one ?
>
> And... isn't there supposed to be some kind of clean-all-ruv task or is that not in 4.2.0 but only a later version ?

More information about the Freeipa-users mailing list