[Freeipa-devel] Topology: Central node removal in star topology

[Ludwig Krispenz]

On 06/24/2015 11:36 AM, Oleg Fayans wrote:
>
>
> On 06/24/2015 11:25 AM, Ludwig Krispenz wrote:
>> Oleg,
>>
>> the topology plugin relies on existing connection between servers 
>> which remain in a topolgy. If you remove a central node in your 
>> topology you are asking for trouble.
>> With Petr's patch it warns you that your topology will be 
>> disconnected, and if you insist we cannot guarantee anything.
> Agree. I just wanted to try edge cases to see how one can break the 
> system :)
>> should we completely prohibit this ? I don't know, I think you could 
>> also enforce an uninstall of vm175 with probably the same result.
>> what you mean be calculating the remaining topology and send it to 
>> the remaining servers does not work, it would require to send a 
>> removal of a segment, which would be rejected.
>>
>> The topology is broken, and I don't know how much we should invest in 
>> making this info consistent on all servers.
>>
>> More interesting would be if we can heal this later by adding new 
>> segments.
> Yes, here comes the biggest question raised from this case: obviously, 
> when none of the nodes possess the correct topology information 
> (including the one which deleted the central node), there is no way to 
> fix it by adding segments connecting the nodes that became disconnected. 
It shoul not need the full information, but it has to be able to reach 
one of the nodes to be connected. when the topology is broken, you loose 
to feature to be ably to apply a change on any node, eg in your case if 
you want to connect vm036 and vm056 an have removed vm175, you have to 
do it on vm056, vm036 or vm244. This should work, if not we have to fix 
it - unless we completely prevent disconnecting a topology
> I still think that the recalculation of the resulting tree should be 
> done at least on the node that performs the removal action. And when 
> later some other node gets connected, it should understand somehow 
> that it's topology information is outdated
>>
>> Ludwig
>> On 06/24/2015 11:04 AM, Oleg Fayans wrote:
>>> Hi everybody,
>>>
>>> Current implementation of topology plugin (including patch 878 from 
>>> Petr) allows the deletion of the central node in the star topology.
>>> I had the following topology:
>>>
>>> vm056      vm036
>>>          \         /     |
>>>          vm175     |
>>>          /         \     |
>>> vm127       vm244
>>>
>>> I was able to remove node vm175 from node vm244:
>>>
>>> [17:54:48]ofayans at vm-244:~]$ ipa-replica-manage del 
>>> vm-175.idm.lab.eng.brq.redhat.com
>>> Topology after removal of vm-175.idm.lab.eng.brq.redhat.com will be 
>>> disconnected:
>>> Server vm-036.idm.lab.eng.brq.redhat.com can't contact servers: 
>>> vm-056.idm.lab.eng.brq.redhat.com, vm-127.idm.lab.eng.brq.redhat.com
>>> Server vm-056.idm.lab.eng.brq.redhat.com can't contact servers: 
>>> vm-244.idm.lab.eng.brq.redhat.com, 
>>> vm-036.idm.lab.eng.brq.redhat.com, vm-127.idm.lab.eng.brq.redhat.com
>>> Server vm-127.idm.lab.eng.brq.redhat.com can't contact servers: 
>>> vm-244.idm.lab.eng.brq.redhat.com, 
>>> vm-056.idm.lab.eng.brq.redhat.com, vm-036.idm.lab.eng.brq.redhat.com
>>> Server vm-244.idm.lab.eng.brq.redhat.com can't contact servers: 
>>> vm-056.idm.lab.eng.brq.redhat.com, vm-127.idm.lab.eng.brq.redhat.com
>>> Continue to delete? [no]: yes
>>> Waiting for removal of replication agreements
>>> unexpected error: limits exceeded for this query
>>>
>>> I would expect this operation to delete 4 replication agreements on 
>>> all nodes:
>>> vm056 - vm175
>>> vm127 - vm175
>>> vm244 - vm175
>>> vm036 - vm175
>>>
>>> However an arbitrary set of replication agreements was deleted on 
>>> each node leading to total infrastructure inconsistency:
>>> ===============================================================
>>> vm056**thought the topology was as follows:
>>> vm056      vm036
>>>                    /     |
>>>          vm175     |
>>>          /         \     |
>>> vm127       vm244
>>> [10:28:55]ofayans at vm-056:~]$ ipa topologysegment-find realm
>>> ------------------
>>> 4 segments matched
>>> ------------------
>>>   Segment name: 036-to-244
>>>   Left node: vm-036.idm.lab.eng.brq.redhat.com
>>>   Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>   Connectivity: both
>>>
>>>   Segment name: 
>>> vm-036.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
>>>   Left node: vm-036.idm.lab.eng.brq.redhat.com
>>>   Right node: vm-175.idm.lab.eng.brq.redhat.com
>>>   Connectivity: both
>>>
>>>   Segment name: 
>>> vm-127.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
>>>   Left node: vm-127.idm.lab.eng.brq.redhat.com
>>>   Right node: vm-175.idm.lab.eng.brq.redhat.com
>>>   Connectivity: both
>>>
>>>   Segment name: 
>>> vm-175.idm.lab.eng.brq.redhat.com-to-vm-244.idm.lab.eng.brq.redhat.com
>>>   Left node: vm-175.idm.lab.eng.brq.redhat.com
>>>   Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>   Connectivity: both
>>> ----------------------------
>>> Number of entries returned 4
>>> ----------------------------
>>> ===============================================================
>>> both vm036**vm244 thought the topology was as follows:
>>> vm056      vm036
>>>          \               |
>>>          vm175     |
>>>          /               |
>>> vm127       vm244
>>>
>>> [10:26:23]ofayans at vm-036:~]$ ipa topologysegment-find
>>> Suffix name: realm
>>> ------------------
>>> 3 segments matched
>>> ------------------
>>>   Segment name: 036-to-244
>>>   Left node: vm-036.idm.lab.eng.brq.redhat.com
>>>   Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>   Connectivity: both
>>>
>>>   Segment name: 
>>> vm-056.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
>>>   Left node: vm-056.idm.lab.eng.brq.redhat.com
>>>   Right node: vm-175.idm.lab.eng.brq.redhat.com
>>>   Connectivity: both
>>>
>>>   Segment name: 
>>> vm-127.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
>>>   Left node: vm-127.idm.lab.eng.brq.redhat.com
>>>   Right node: vm-175.idm.lab.eng.brq.redhat.com
>>>   Connectivity: both
>>> ----------------------------
>>> Number of entries returned 3
>>> ----------------------------
>>>
>>> ===============================================================
>>> **vm127 thought the topology was as follows:
>>> vm056      vm036
>>>          \        /      |
>>>          vm175     |
>>>                   \      |
>>> vm127       vm244
>>>
>>> [10:31:08]ofayans at vm-127:~]$ ipa topologysegment-find realm
>>> ------------------
>>> 4 segments matched
>>> ------------------
>>>   Segment name: 036-to-244
>>>   Left node: vm-036.idm.lab.eng.brq.redhat.com
>>>   Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>   Connectivity: both
>>>
>>>   Segment name: 
>>> vm-036.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
>>>   Left node: vm-036.idm.lab.eng.brq.redhat.com
>>>   Right node: vm-175.idm.lab.eng.brq.redhat.com
>>>   Connectivity: both
>>>
>>>   Segment name: 
>>> vm-056.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
>>>   Left node: vm-056.idm.lab.eng.brq.redhat.com
>>>   Right node: vm-175.idm.lab.eng.brq.redhat.com
>>>   Connectivity: both
>>>
>>>   Segment name: 
>>> vm-175.idm.lab.eng.brq.redhat.com-to-vm-244.idm.lab.eng.brq.redhat.com
>>>   Left node: vm-175.idm.lab.eng.brq.redhat.com
>>>   Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>   Connectivity: both
>>> ----------------------------
>>> Number of entries returned 4
>>> ----------------------------
>>>
>>> If I, for example, add a segment connecting vm127 and vm244, these 
>>> two nodes will not synchronize the topology info:
>>>
>>> [10:51:03]ofayans at vm-127:~]$ ipa topologysegment-add realm 
>>> 127-to-244 --leftnode=vm-127.idm.lab.eng.brq.redhat.com 
>>> --rightnode=vm-244.idm.lab.eng.brq.redhat.com --direction=both
>>> --------------------------
>>> Added segment "127-to-244"
>>> --------------------------
>>>   Segment name: 127-to-244
>>>   Left node: vm-127.idm.lab.eng.brq.redhat.com
>>>   Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>   Connectivity: both
>>> [10:53:33]ofayans at vm-127:~]$ ipa topologysegment-find realm
>>> ------------------
>>> 5 segments matched
>>> ------------------
>>>   Segment name: 036-to-244
>>>   Left node: vm-036.idm.lab.eng.brq.redhat.com
>>>   Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>   Connectivity: both
>>>
>>>   Segment name: 127-to-244
>>>   Left node: vm-127.idm.lab.eng.brq.redhat.com
>>>   Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>   Connectivity: both
>>>
>>>   Segment name: 
>>> vm-036.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
>>>   Left node: vm-036.idm.lab.eng.brq.redhat.com
>>>   Right node: vm-175.idm.lab.eng.brq.redhat.com
>>>   Connectivity: both
>>>
>>>   Segment name: 
>>> vm-056.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
>>>   Left node: vm-056.idm.lab.eng.brq.redhat.com
>>>   Right node: vm-175.idm.lab.eng.brq.redhat.com
>>>   Connectivity: both
>>>
>>>   Segment name: 
>>> vm-175.idm.lab.eng.brq.redhat.com-to-vm-244.idm.lab.eng.brq.redhat.com
>>>   Left node: vm-175.idm.lab.eng.brq.redhat.com
>>>   Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>   Connectivity: both
>>> ----------------------------
>>> Number of entries returned 5
>>> ----------------------------
>>> [10:54:02]ofayans at vm-127:~]$
>>>
>>> =============================================================
>>>
>>> [10:49:38]ofayans at vm-244:~]$ ipa topologysegment-find realm
>>> ------------------
>>> 3 segments matched
>>> ------------------
>>>   Segment name: 036-to-244
>>>   Left node: vm-036.idm.lab.eng.brq.redhat.com
>>>   Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>   Connectivity: both
>>>
>>>   Segment name: 127-to-244
>>>   Left node: vm-127.idm.lab.eng.brq.redhat.com
>>>   Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>   Connectivity: both
>>>
>>>   Segment name: 
>>> vm-056.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
>>>   Left node: vm-056.idm.lab.eng.brq.redhat.com
>>>   Right node: vm-175.idm.lab.eng.brq.redhat.com
>>>   Connectivity: both
>>> ----------------------------
>>> Number of entries returned 3
>>> ----------------------------
>>> [10:56:34]ofayans at vm-244:~]$
>>>
>>> Conclusion:
>>> We either should completely prohibit the removal of the middle nodes 
>>> (I mean, nodes that hide another active nodes),
>>> or at the removal stage first recalculate the resulting topology and 
>>> send it to all nodes before actual removal.
>>> -- 
>>> Oleg Fayans
>>> Quality Engineer
>>> FreeIPA team
>>> RedHat.
>>>
>>>
>>
>>
>>
>
> -- 
> Oleg Fayans
> Quality Engineer
> FreeIPA team
> RedHat.
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/freeipa-devel/attachments/20150624/107e974b/attachment.htm>