[Freeipa-users] clean-ruv

Wed Aug 24 13:48:01 UTC 2016

On 08/23/2016 05:52 AM, Ian Harding wrote:
> Ah.  I see.  I mixed those up but I see that those would have to be
> consistent.
>
> However, I have been trying to beat some invalid RUV to death for a long
> time and I can't seem to kill them.
>
> For example, bellevuenfs has 9 and 16 which are invalid:
>
> [ianh at seattlenfs ~]$ ldapsearch -ZZ -h seattlenfs.bpt.rocks -D
> "cn=Directory Manager" -W -b "dc=bpt,dc=rocks"
> "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))"
> | grep "nsds50ruv\|nsDS5ReplicaId"
> Enter LDAP Password:
> nsDS5ReplicaId: 7
> nsds50ruv: {replicageneration} 55c8f364000000040000
> nsds50ruv: {replica 7 ldap://seattlenfs.bpt.rocks:389}
> 568ac3cc000000070000 57
> nsds50ruv: {replica 20 ldap://freeipa-sea.bpt.rocks:389}
> 57b10377000200140000
> nsds50ruv: {replica 18 ldap://bpt-nyc1-nfs.bpt.rocks:389}
> 57a47801000100120000
> nsds50ruv: {replica 15 ldap://fremontnis.bpt.rocks:389}
> 57a403860000000f0000 5
> nsds50ruv: {replica 14 ldap://freeipa-dal.bpt.rocks:389}
> 57a2dccd0000000e0000
> nsds50ruv: {replica 17 ldap://edinburghnfs.bpt.rocks:389}
> 57a422f9000000110000
> nsds50ruv: {replica 19 ldap://bellevuenfs.bpt.rocks:389}
> 57a4f20d000600130000
> nsds50ruv: {replica 16 ldap://bellevuenfs.bpt.rocks:389}
> 57a41706000000100000
> nsds50ruv: {replica 9 ldap://bellevuenfs.bpt.rocks:389}
> 570484ee000000090000 5
>
>
> So I try to kill them like so:
> [ianh at seattlenfs ~]$ ipa-replica-manage clean-ruv 9 --force --cleanup
> ipa: WARNING: session memcached servers not running
> Clean the Replication Update Vector for bellevuenfs.bpt.rocks:389
>
> Cleaning the wrong replica ID will cause that server to no
> longer replicate so it may miss updates while the process
> is running. It would need to be re-initialized to maintain
> consistency. Be very careful.
> Background task created to clean replication data. This may take a while.
> This may be safely interrupted with Ctrl+C
> ^C[ianh at seattlenfs ~]$ ipa-replica-manage clean-ruv 16 --force --cleanup
> ipa: WARNING: session memcached servers not running
> Clean the Replication Update Vector for bellevuenfs.bpt.rocks:389
>
> Cleaning the wrong replica ID will cause that server to no
> longer replicate so it may miss updates while the process
> is running. It would need to be re-initialized to maintain
> consistency. Be very careful.
> Background task created to clean replication data. This may take a while.
> This may be safely interrupted with Ctrl+C
> ^C[ianh at seattlenfs ~]$ ipa-replica-manage list-clean-ruv
> ipa: WARNING: session memcached servers not running
> CLEANALLRUV tasks
> RID 16: Waiting to process all the updates from the deleted replica...
> RID 9: Waiting to process all the updates from the deleted replica...
Looks like you are hitting a bug that is fixed in newer versions of
389-ds-base.  The current version of 389-ds-base/cleanAllRUV does not
wait for updates from the deleted replica if you use the force option. 
Since you did use the force option, and it's still waiting, that tells
me you are hitting this old bug and ultimately you need to upgrade or
get a hotfix(if you are paying customer). 

I do not know what version of 389 you are on, or if it's possible to
upgrade, but with your current version the cleanAllRUV task is not going
to be able to finish.

You can always "abort" the current "clean"  tasks that are not working
(look for the abort section from the link below), but unfortunately you
won't be able to clean those rids until you upgrade 389-ds-base.

http://www.port389.org/docs/389ds/howto/howto-cleanruv.html#cleanallruv

Regards,
Mark
>
> No abort CLEANALLRUV tasks running
> [ianh at seattlenfs ~]$ ipa-replica-manage list-clean-ruv
> ipa: WARNING: session memcached servers not running
> CLEANALLRUV tasks
> RID 16: Waiting to process all the updates from the deleted replica...
> RID 9: Waiting to process all the updates from the deleted replica...
>
> and it never finishes.
>
> seattlenfs is the first master, that's the only place I should have to
> run this command, right?
>
> I'm about to burn everything down and ipa-server-install --uninstall but
> I've done that before a couple times and that seems to be what got me
> into this mess...
>
> Thank you for your help.
>
>
>
>
> On 08/23/2016 01:37 AM, Ludwig Krispenz wrote:
>> looks like you are searching the nstombstone below "o=ipaca", but you
>> are cleaning ruvs in "dc=bpt,dc=rocks",
>>
>> your attrlist_replace error refers to the bpt,rocks backend, so you
>> should search the tombstone entry ther, then determine which replicaIDs
>> to remove.
>>
>> Ludwig
>>
>> On 08/23/2016 09:20 AM, Ian Harding wrote:
>>> I've followed the procedure in this thread:
>>>
>>> https://www.redhat.com/archives/freeipa-users/2016-May/msg00043.html
>>>
>>> and found my list of RUV that don't have an existing replica id.
>>>
>>> I've tried to remove them like so:
>>>
>>> [root at seattlenfs ianh]# ldapmodify -D "cn=directory manager" -W -a
>>> Enter LDAP Password:
>>> dn: cn=clean 97, cn=cleanallruv, cn=tasks, cn=config
>>> objectclass: top
>>> objectclass: extensibleObject
>>> replica-base-dn: dc=bpt,dc=rocks
>>> replica-id: 97
>>> replica-force-cleaning: yes
>>> cn: clean 97
>>>
>>> adding new entry "cn=clean 97, cn=cleanallruv, cn=tasks, cn=config"
>>>
>>> [root at seattlenfs ianh]# ipa-replica-manage list-clean-ruv
>>> CLEANALLRUV tasks
>>> RID 9: Waiting to process all the updates from the deleted replica...
>>> RID 96: Successfully cleaned rid(96).
>>> RID 97: Successfully cleaned rid(97).
>>>
>>> No abort CLEANALLRUV tasks running
>>>
>>>
>>> and yet, they are still there...
>>>
>>> [root at seattlenfs ianh]# ldapsearch -ZZ -h seattlenfs.bpt.rocks -D
>>> "cn=Directory Manager" -W -b "o=ipaca"
>>> "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))"
>>>
>>> | grep "nsds50ruv\|nsDS5ReplicaId"
>>> Enter LDAP Password:
>>> nsDS5ReplicaId: 81
>>> nsds50ruv: {replicageneration} 55c8f3ae000000600000
>>> nsds50ruv: {replica 81 ldap://seattlenfs.bpt.rocks:389}
>>> 568ac431000000510000 5
>>> nsds50ruv: {replica 1065 ldap://freeipa-sea.bpt.rocks:389}
>>> 57b103d400000429000
>>> nsds50ruv: {replica 1070 ldap://bellevuenfs.bpt.rocks:389}
>>> 57a4f2700000042e000
>>> nsds50ruv: {replica 1075 ldap://bpt-nyc1-nfs.bpt.rocks:389}
>>> 57a478650000043300
>>> nsds50ruv: {replica 1080 ldap://bellevuenfs.bpt.rocks:389}
>>> 57a4176700000438000
>>> nsds50ruv: {replica 1085 ldap://fremontnis.bpt.rocks:389}
>>> 57a403e60000043d0000
>>> nsds50ruv: {replica 1090 ldap://freeipa-dal.bpt.rocks:389}
>>> 57a2dd3500000442000
>>> nsds50ruv: {replica 1095 ldap://freeipa-sea.bpt.rocks:389}
>>> 579a963c00000447000
>>> nsds50ruv: {replica 96 ldap://freeipa-sea.bpt.rocks:389}
>>> 55c8f3bd000000600000
>>> nsds50ruv: {replica 86 ldap://fremontnis.bpt.rocks:389}
>>> 5685b24e000000560000 5
>>> nsds50ruv: {replica 91 ldap://seattlenis.bpt.rocks:389}
>>> 567ad6180001005b0000 5
>>> nsds50ruv: {replica 97 ldap://freeipa-dal.bpt.rocks:389}
>>> 55c8f3ce000000610000
>>> nsds50ruv: {replica 76 ldap://bellevuenis.bpt.rocks:389}
>>> 56f385eb0007004c0000
>>> nsds50ruv: {replica 71 ldap://bellevuenfs.bpt.rocks:389}
>>> 57048560000900470000
>>> nsds50ruv: {replica 66 ldap://bpt-nyc1-nfs.bpt.rocks:389}
>>> 5733e594000a00420000
>>> nsds50ruv: {replica 61 ldap://edinburghnfs.bpt.rocks:389}
>>> 574421250000003d0000
>>> nsds50ruv: {replica 1195 ldap://edinburghnfs.bpt.rocks:389}
>>> 57a42390000004ab00
>>>
>>> What have I done wrong?
>>>
>>> The problem I am trying to solve is that seattlenfs.bpt.rocks sends
>>> updates to all its children, but their changes don't come back because
>>> of these errors:
>>>
>>> [23/Aug/2016:00:02:16 -0700] attrlist_replace - attr_replace
>>> (nsslapd-referral,
>>> ldap://seattlenfs.bpt.rocks:389/dc%3Dbpt%2Cdc%3Drocks) failed.
>>>
>>> in effect, the replication agreements are one-way.
>>>
>>> Any ideas?
>>>
>>> - Ian
>>>