[Freeipa-devel] Topology: Central node removal in star topology

[Ludwig Krispenz]

On 06/24/2015 09:01 PM, Simo Sorce wrote:
> On Wed, 2015-06-24 at 11:25 +0200, Ludwig Krispenz wrote:
>> Oleg,
>>
>> the topology plugin relies on existing connection between servers which
>> remain in a topolgy. If you remove a central node in your topology you
>> are asking for trouble.
>> With Petr's patch it warns you that your topology will be disconnected,
>> and if you insist we cannot guarantee anything.
>> should we completely prohibit this ?
> No, but a --force should be needed.
> Without a --force option we should not allow to remove a replica
> completely from another one.
>
>> I don't know, I think you could
>> also enforce an uninstall of vm175 with probably the same result.
>> what you mean be calculating the remaining topology and send it to the
>> remaining servers does not work, it would require to send a removal of a
>> segment, which would be rejected.
> You would have to connect to each replica that has a replication
> agreement with vm175 and remove the segment from that replica. But it
> wouldn't really help much as once a replica is isolated from the central
> one, it will not see the other operations going on in other replicas.
>
> Once we have a topology resolver we will be able to warn that removing a
> specific replica will cause a split brain and make very loud warnings
we have this already, see the output of Oleg's example:

ipa-replica-manage del vm-175.idm.lab.eng.brq.redhat.com
Topology after removal of vm-175.idm.lab.eng.brq.redhat.com will be 
disconnected:
Server vm-036.idm.lab.eng.brq.redhat.com can't contact servers: 
vm-056.idm.lab.eng.brq.redhat.com, vm-127.idm.lab.eng.brq.redhat.com
Server vm-056.idm.lab.eng.brq.redhat.com can't contact servers: 
vm-244.idm.lab.eng.brq.redhat.com, vm-036.idm.lab.eng.brq.redhat.com, 
vm-127.idm.lab.eng.brq.redhat.com
Server vm-127.idm.lab.eng.brq.redhat.com can't contact servers: 
vm-244.idm.lab.eng.brq.redhat.com, vm-056.idm.lab.eng.brq.redhat.com, 
vm-036.idm.lab.eng.brq.redhat.com
Server vm-244.idm.lab.eng.brq.redhat.com can't contact servers: 
vm-056.idm.lab.eng.brq.redhat.com, vm-127.idm.lab.eng.brq.redhat.com
Continue to delete? [no]: yes

it tells you that the topology gets disconnected and which connections 
will be missing, the continue yes/no is the --force,
the question was, should we allow a force in this situation ?

>> More interesting would be if we can heal this later by adding new segments.
> Indeed, reconnecting all the severed replicas should cause all the
> removals (segments or servers) to be replicated among servers and should
> bring back the topology view in a consistent state. But not until all
> servers are reconnected and replication has started again.
This healing can also be required without forcing removal by an admin. 
If you have a start topology and your central node goes down and is not 
recoverable
>
> Simo.
>
>
>> Ludwig
>> On 06/24/2015 11:04 AM, Oleg Fayans wrote:
>>> Hi everybody,
>>>
>>> Current implementation of topology plugin (including patch 878 from
>>> Petr) allows the deletion of the central node in the star topology.
>>> I had the following topology:
>>>
>>> vm056      vm036
>>>           \         /     |
>>>           vm175     |
>>>           /         \     |
>>> vm127       vm244
>>>
>>> I was able to remove node vm175 from node vm244:
>>>
>>> [17:54:48]ofayans at vm-244:~]$ ipa-replica-manage del
>>> vm-175.idm.lab.eng.brq.redhat.com
>>> Topology after removal of vm-175.idm.lab.eng.brq.redhat.com will be
>>> disconnected:
>>> Server vm-036.idm.lab.eng.brq.redhat.com can't contact servers:
>>> vm-056.idm.lab.eng.brq.redhat.com, vm-127.idm.lab.eng.brq.redhat.com
>>> Server vm-056.idm.lab.eng.brq.redhat.com can't contact servers:
>>> vm-244.idm.lab.eng.brq.redhat.com, vm-036.idm.lab.eng.brq.redhat.com,
>>> vm-127.idm.lab.eng.brq.redhat.com
>>> Server vm-127.idm.lab.eng.brq.redhat.com can't contact servers:
>>> vm-244.idm.lab.eng.brq.redhat.com, vm-056.idm.lab.eng.brq.redhat.com,
>>> vm-036.idm.lab.eng.brq.redhat.com
>>> Server vm-244.idm.lab.eng.brq.redhat.com can't contact servers:
>>> vm-056.idm.lab.eng.brq.redhat.com, vm-127.idm.lab.eng.brq.redhat.com
>>> Continue to delete? [no]: yes
>>> Waiting for removal of replication agreements
>>> unexpected error: limits exceeded for this query
>>>
>>> I would expect this operation to delete 4 replication agreements on
>>> all nodes:
>>> vm056 - vm175
>>> vm127 - vm175
>>> vm244 - vm175
>>> vm036 - vm175
>>>
>>> However an arbitrary set of replication agreements was deleted on each
>>> node leading to total infrastructure inconsistency:
>>> ===============================================================
>>> vm056**thought the topology was as follows:
>>> vm056      vm036
>>>                     /     |
>>>           vm175     |
>>>           /         \     |
>>> vm127       vm244
>>> [10:28:55]ofayans at vm-056:~]$ ipa topologysegment-find realm
>>> ------------------
>>> 4 segments matched
>>> ------------------
>>>    Segment name: 036-to-244
>>>    Left node: vm-036.idm.lab.eng.brq.redhat.com
>>>    Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>    Connectivity: both
>>>
>>>    Segment name:
>>> vm-036.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
>>>    Left node: vm-036.idm.lab.eng.brq.redhat.com
>>>    Right node: vm-175.idm.lab.eng.brq.redhat.com
>>>    Connectivity: both
>>>
>>>    Segment name:
>>> vm-127.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
>>>    Left node: vm-127.idm.lab.eng.brq.redhat.com
>>>    Right node: vm-175.idm.lab.eng.brq.redhat.com
>>>    Connectivity: both
>>>
>>>    Segment name:
>>> vm-175.idm.lab.eng.brq.redhat.com-to-vm-244.idm.lab.eng.brq.redhat.com
>>>    Left node: vm-175.idm.lab.eng.brq.redhat.com
>>>    Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>    Connectivity: both
>>> ----------------------------
>>> Number of entries returned 4
>>> ----------------------------
>>> ===============================================================
>>> both vm036**vm244 thought the topology was as follows:
>>> vm056      vm036
>>>           \               |
>>>           vm175     |
>>>           /               |
>>> vm127       vm244
>>>
>>> [10:26:23]ofayans at vm-036:~]$ ipa topologysegment-find
>>> Suffix name: realm
>>> ------------------
>>> 3 segments matched
>>> ------------------
>>>    Segment name: 036-to-244
>>>    Left node: vm-036.idm.lab.eng.brq.redhat.com
>>>    Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>    Connectivity: both
>>>
>>>    Segment name:
>>> vm-056.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
>>>    Left node: vm-056.idm.lab.eng.brq.redhat.com
>>>    Right node: vm-175.idm.lab.eng.brq.redhat.com
>>>    Connectivity: both
>>>
>>>    Segment name:
>>> vm-127.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
>>>    Left node: vm-127.idm.lab.eng.brq.redhat.com
>>>    Right node: vm-175.idm.lab.eng.brq.redhat.com
>>>    Connectivity: both
>>> ----------------------------
>>> Number of entries returned 3
>>> ----------------------------
>>>
>>> ===============================================================
>>> **vm127 thought the topology was as follows:
>>> vm056      vm036
>>>           \        /      |
>>>           vm175     |
>>>                    \      |
>>> vm127       vm244
>>>
>>> [10:31:08]ofayans at vm-127:~]$ ipa topologysegment-find realm
>>> ------------------
>>> 4 segments matched
>>> ------------------
>>>    Segment name: 036-to-244
>>>    Left node: vm-036.idm.lab.eng.brq.redhat.com
>>>    Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>    Connectivity: both
>>>
>>>    Segment name:
>>> vm-036.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
>>>    Left node: vm-036.idm.lab.eng.brq.redhat.com
>>>    Right node: vm-175.idm.lab.eng.brq.redhat.com
>>>    Connectivity: both
>>>
>>>    Segment name:
>>> vm-056.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
>>>    Left node: vm-056.idm.lab.eng.brq.redhat.com
>>>    Right node: vm-175.idm.lab.eng.brq.redhat.com
>>>    Connectivity: both
>>>
>>>    Segment name:
>>> vm-175.idm.lab.eng.brq.redhat.com-to-vm-244.idm.lab.eng.brq.redhat.com
>>>    Left node: vm-175.idm.lab.eng.brq.redhat.com
>>>    Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>    Connectivity: both
>>> ----------------------------
>>> Number of entries returned 4
>>> ----------------------------
>>>
>>> If I, for example, add a segment connecting vm127 and vm244, these two
>>> nodes will not synchronize the topology info:
>>>
>>> [10:51:03]ofayans at vm-127:~]$ ipa topologysegment-add realm 127-to-244
>>> --leftnode=vm-127.idm.lab.eng.brq.redhat.com
>>> --rightnode=vm-244.idm.lab.eng.brq.redhat.com --direction=both
>>> --------------------------
>>> Added segment "127-to-244"
>>> --------------------------
>>>    Segment name: 127-to-244
>>>    Left node: vm-127.idm.lab.eng.brq.redhat.com
>>>    Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>    Connectivity: both
>>> [10:53:33]ofayans at vm-127:~]$ ipa topologysegment-find realm
>>> ------------------
>>> 5 segments matched
>>> ------------------
>>>    Segment name: 036-to-244
>>>    Left node: vm-036.idm.lab.eng.brq.redhat.com
>>>    Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>    Connectivity: both
>>>
>>>    Segment name: 127-to-244
>>>    Left node: vm-127.idm.lab.eng.brq.redhat.com
>>>    Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>    Connectivity: both
>>>
>>>    Segment name:
>>> vm-036.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
>>>    Left node: vm-036.idm.lab.eng.brq.redhat.com
>>>    Right node: vm-175.idm.lab.eng.brq.redhat.com
>>>    Connectivity: both
>>>
>>>    Segment name:
>>> vm-056.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
>>>    Left node: vm-056.idm.lab.eng.brq.redhat.com
>>>    Right node: vm-175.idm.lab.eng.brq.redhat.com
>>>    Connectivity: both
>>>
>>>    Segment name:
>>> vm-175.idm.lab.eng.brq.redhat.com-to-vm-244.idm.lab.eng.brq.redhat.com
>>>    Left node: vm-175.idm.lab.eng.brq.redhat.com
>>>    Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>    Connectivity: both
>>> ----------------------------
>>> Number of entries returned 5
>>> ----------------------------
>>> [10:54:02]ofayans at vm-127:~]$
>>>
>>> =============================================================
>>>
>>> [10:49:38]ofayans at vm-244:~]$ ipa topologysegment-find realm
>>> ------------------
>>> 3 segments matched
>>> ------------------
>>>    Segment name: 036-to-244
>>>    Left node: vm-036.idm.lab.eng.brq.redhat.com
>>>    Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>    Connectivity: both
>>>
>>>    Segment name: 127-to-244
>>>    Left node: vm-127.idm.lab.eng.brq.redhat.com
>>>    Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>    Connectivity: both
>>>
>>>    Segment name:
>>> vm-056.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
>>>    Left node: vm-056.idm.lab.eng.brq.redhat.com
>>>    Right node: vm-175.idm.lab.eng.brq.redhat.com
>>>    Connectivity: both
>>> ----------------------------
>>> Number of entries returned 3
>>> ----------------------------
>>> [10:56:34]ofayans at vm-244:~]$
>>>
>>> Conclusion:
>>> We either should completely prohibit the removal of the middle nodes
>>> (I mean, nodes that hide another active nodes),
>>> or at the removal stage first recalculate the resulting topology and
>>> send it to all nodes before actual removal.
>>> -- 
>>> Oleg Fayans
>>> Quality Engineer
>>> FreeIPA team
>>> RedHat.
>>>
>>>
>