[Freeipa-users] clean-ruv

Tue Aug 23 09:52:42 UTC 2016

Ah.  I see.  I mixed those up but I see that those would have to be
consistent.

However, I have been trying to beat some invalid RUV to death for a long
time and I can't seem to kill them.

For example, bellevuenfs has 9 and 16 which are invalid:

[ianh at seattlenfs ~]$ ldapsearch -ZZ -h seattlenfs.bpt.rocks -D
"cn=Directory Manager" -W -b "dc=bpt,dc=rocks"
"(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))"
| grep "nsds50ruv\|nsDS5ReplicaId"
Enter LDAP Password:
nsDS5ReplicaId: 7
nsds50ruv: {replicageneration} 55c8f364000000040000
nsds50ruv: {replica 7 ldap://seattlenfs.bpt.rocks:389}
568ac3cc000000070000 57
nsds50ruv: {replica 20 ldap://freeipa-sea.bpt.rocks:389}
57b10377000200140000
nsds50ruv: {replica 18 ldap://bpt-nyc1-nfs.bpt.rocks:389}
57a47801000100120000
nsds50ruv: {replica 15 ldap://fremontnis.bpt.rocks:389}
57a403860000000f0000 5
nsds50ruv: {replica 14 ldap://freeipa-dal.bpt.rocks:389}
57a2dccd0000000e0000
nsds50ruv: {replica 17 ldap://edinburghnfs.bpt.rocks:389}
57a422f9000000110000
nsds50ruv: {replica 19 ldap://bellevuenfs.bpt.rocks:389}
57a4f20d000600130000
nsds50ruv: {replica 16 ldap://bellevuenfs.bpt.rocks:389}
57a41706000000100000
nsds50ruv: {replica 9 ldap://bellevuenfs.bpt.rocks:389}
570484ee000000090000 5

So I try to kill them like so:
[ianh at seattlenfs ~]$ ipa-replica-manage clean-ruv 9 --force --cleanup
ipa: WARNING: session memcached servers not running
Clean the Replication Update Vector for bellevuenfs.bpt.rocks:389

Cleaning the wrong replica ID will cause that server to no
longer replicate so it may miss updates while the process
is running. It would need to be re-initialized to maintain
consistency. Be very careful.
Background task created to clean replication data. This may take a while.
This may be safely interrupted with Ctrl+C
^C[ianh at seattlenfs ~]$ ipa-replica-manage clean-ruv 16 --force --cleanup
ipa: WARNING: session memcached servers not running
Clean the Replication Update Vector for bellevuenfs.bpt.rocks:389

Cleaning the wrong replica ID will cause that server to no
longer replicate so it may miss updates while the process
is running. It would need to be re-initialized to maintain
consistency. Be very careful.
Background task created to clean replication data. This may take a while.
This may be safely interrupted with Ctrl+C
^C[ianh at seattlenfs ~]$ ipa-replica-manage list-clean-ruv
ipa: WARNING: session memcached servers not running
CLEANALLRUV tasks
RID 16: Waiting to process all the updates from the deleted replica...
RID 9: Waiting to process all the updates from the deleted replica...

No abort CLEANALLRUV tasks running
[ianh at seattlenfs ~]$ ipa-replica-manage list-clean-ruv
ipa: WARNING: session memcached servers not running
CLEANALLRUV tasks
RID 16: Waiting to process all the updates from the deleted replica...
RID 9: Waiting to process all the updates from the deleted replica...

and it never finishes.

seattlenfs is the first master, that's the only place I should have to
run this command, right?

I'm about to burn everything down and ipa-server-install --uninstall but
I've done that before a couple times and that seems to be what got me
into this mess...

Thank you for your help.

On 08/23/2016 01:37 AM, Ludwig Krispenz wrote:
> looks like you are searching the nstombstone below "o=ipaca", but you
> are cleaning ruvs in "dc=bpt,dc=rocks",
> 
> your attrlist_replace error refers to the bpt,rocks backend, so you
> should search the tombstone entry ther, then determine which replicaIDs
> to remove.
> 
> Ludwig
> 
> On 08/23/2016 09:20 AM, Ian Harding wrote:
>> I've followed the procedure in this thread:
>>
>> https://www.redhat.com/archives/freeipa-users/2016-May/msg00043.html
>>
>> and found my list of RUV that don't have an existing replica id.
>>
>> I've tried to remove them like so:
>>
>> [root at seattlenfs ianh]# ldapmodify -D "cn=directory manager" -W -a
>> Enter LDAP Password:
>> dn: cn=clean 97, cn=cleanallruv, cn=tasks, cn=config
>> objectclass: top
>> objectclass: extensibleObject
>> replica-base-dn: dc=bpt,dc=rocks
>> replica-id: 97
>> replica-force-cleaning: yes
>> cn: clean 97
>>
>> adding new entry "cn=clean 97, cn=cleanallruv, cn=tasks, cn=config"
>>
>> [root at seattlenfs ianh]# ipa-replica-manage list-clean-ruv
>> CLEANALLRUV tasks
>> RID 9: Waiting to process all the updates from the deleted replica...
>> RID 96: Successfully cleaned rid(96).
>> RID 97: Successfully cleaned rid(97).
>>
>> No abort CLEANALLRUV tasks running
>>
>>
>> and yet, they are still there...
>>
>> [root at seattlenfs ianh]# ldapsearch -ZZ -h seattlenfs.bpt.rocks -D
>> "cn=Directory Manager" -W -b "o=ipaca"
>> "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))"
>>
>> | grep "nsds50ruv\|nsDS5ReplicaId"
>> Enter LDAP Password:
>> nsDS5ReplicaId: 81
>> nsds50ruv: {replicageneration} 55c8f3ae000000600000
>> nsds50ruv: {replica 81 ldap://seattlenfs.bpt.rocks:389}
>> 568ac431000000510000 5
>> nsds50ruv: {replica 1065 ldap://freeipa-sea.bpt.rocks:389}
>> 57b103d400000429000
>> nsds50ruv: {replica 1070 ldap://bellevuenfs.bpt.rocks:389}
>> 57a4f2700000042e000
>> nsds50ruv: {replica 1075 ldap://bpt-nyc1-nfs.bpt.rocks:389}
>> 57a478650000043300
>> nsds50ruv: {replica 1080 ldap://bellevuenfs.bpt.rocks:389}
>> 57a4176700000438000
>> nsds50ruv: {replica 1085 ldap://fremontnis.bpt.rocks:389}
>> 57a403e60000043d0000
>> nsds50ruv: {replica 1090 ldap://freeipa-dal.bpt.rocks:389}
>> 57a2dd3500000442000
>> nsds50ruv: {replica 1095 ldap://freeipa-sea.bpt.rocks:389}
>> 579a963c00000447000
>> nsds50ruv: {replica 96 ldap://freeipa-sea.bpt.rocks:389}
>> 55c8f3bd000000600000
>> nsds50ruv: {replica 86 ldap://fremontnis.bpt.rocks:389}
>> 5685b24e000000560000 5
>> nsds50ruv: {replica 91 ldap://seattlenis.bpt.rocks:389}
>> 567ad6180001005b0000 5
>> nsds50ruv: {replica 97 ldap://freeipa-dal.bpt.rocks:389}
>> 55c8f3ce000000610000
>> nsds50ruv: {replica 76 ldap://bellevuenis.bpt.rocks:389}
>> 56f385eb0007004c0000
>> nsds50ruv: {replica 71 ldap://bellevuenfs.bpt.rocks:389}
>> 57048560000900470000
>> nsds50ruv: {replica 66 ldap://bpt-nyc1-nfs.bpt.rocks:389}
>> 5733e594000a00420000
>> nsds50ruv: {replica 61 ldap://edinburghnfs.bpt.rocks:389}
>> 574421250000003d0000
>> nsds50ruv: {replica 1195 ldap://edinburghnfs.bpt.rocks:389}
>> 57a42390000004ab00
>>
>> What have I done wrong?
>>
>> The problem I am trying to solve is that seattlenfs.bpt.rocks sends
>> updates to all its children, but their changes don't come back because
>> of these errors:
>>
>> [23/Aug/2016:00:02:16 -0700] attrlist_replace - attr_replace
>> (nsslapd-referral,
>> ldap://seattlenfs.bpt.rocks:389/dc%3Dbpt%2Cdc%3Drocks) failed.
>>
>> in effect, the replication agreements are one-way.
>>
>> Any ideas?
>>
>> - Ian
>>
> 

-- 
Ian Harding
IT Director
Brown Paper Tickets
1-800-838-3006 ext 7186
http://www.brownpapertickets.com