[Freeipa-devel] Topology: Central node removal in star topology
Oleg Fayans
ofayans at redhat.com
Wed Jun 24 10:02:06 UTC 2015
On 06/24/2015 11:47 AM, Ludwig Krispenz wrote:
>
> On 06/24/2015 11:36 AM, Oleg Fayans wrote:
>>
>>
>> On 06/24/2015 11:25 AM, Ludwig Krispenz wrote:
>>> Oleg,
>>>
>>> the topology plugin relies on existing connection between servers
>>> which remain in a topolgy. If you remove a central node in your
>>> topology you are asking for trouble.
>>> With Petr's patch it warns you that your topology will be
>>> disconnected, and if you insist we cannot guarantee anything.
>> Agree. I just wanted to try edge cases to see how one can break the
>> system :)
>>> should we completely prohibit this ? I don't know, I think you could
>>> also enforce an uninstall of vm175 with probably the same result.
>>> what you mean be calculating the remaining topology and send it to
>>> the remaining servers does not work, it would require to send a
>>> removal of a segment, which would be rejected.
>>>
>>> The topology is broken, and I don't know how much we should invest
>>> in making this info consistent on all servers.
>>>
>>> More interesting would be if we can heal this later by adding new
>>> segments.
>> Yes, here comes the biggest question raised from this case:
>> obviously, when none of the nodes possess the correct topology
>> information (including the one which deleted the central node), there
>> is no way to fix it by adding segments connecting the nodes that
>> became disconnected.
> It shoul not need the full information, but it has to be able to reach
> one of the nodes to be connected. when the topology is broken, you
> loose to feature to be ably to apply a change on any node, eg in your
> case if you want to connect vm036 and vm056 an have removed vm175, you
> have to do it on vm056, vm036 or vm244. This should work, if not we
> have to fix it - unless we completely prevent disconnecting a topology
Well, this is exactly the problem here: all replicas should contain
precise copies of all the info: accounts, hosts, sudorules, etc,
including topology information. However, if in this case I manually
connect disconnected node at vm127 (or vm056, does not matter) it
results in topology information inconsistency across the infrastructure:
This would be the topology from the point of view of vm127:
vm056 vm036
\ / |
vm175 |
\ |
vm127 vm244
And this - from the point of view of vm244 and vm036
vm056 vm036
\ |
vm175 |
|
vm127 ----- vm244
>> I still think that the recalculation of the resulting tree should be
>> done at least on the node that performs the removal action. And when
>> later some other node gets connected, it should understand somehow
>> that it's topology information is outdated
>>>
>>> Ludwig
>>> On 06/24/2015 11:04 AM, Oleg Fayans wrote:
>>>> Hi everybody,
>>>>
>>>> Current implementation of topology plugin (including patch 878 from
>>>> Petr) allows the deletion of the central node in the star topology.
>>>> I had the following topology:
>>>>
>>>> vm056 vm036
>>>> \ / |
>>>> vm175 |
>>>> / \ |
>>>> vm127 vm244
>>>>
>>>> I was able to remove node vm175 from node vm244:
>>>>
>>>> [17:54:48]ofayans at vm-244:~]$ ipa-replica-manage del
>>>> vm-175.idm.lab.eng.brq.redhat.com
>>>> Topology after removal of vm-175.idm.lab.eng.brq.redhat.com will be
>>>> disconnected:
>>>> Server vm-036.idm.lab.eng.brq.redhat.com can't contact servers:
>>>> vm-056.idm.lab.eng.brq.redhat.com, vm-127.idm.lab.eng.brq.redhat.com
>>>> Server vm-056.idm.lab.eng.brq.redhat.com can't contact servers:
>>>> vm-244.idm.lab.eng.brq.redhat.com,
>>>> vm-036.idm.lab.eng.brq.redhat.com, vm-127.idm.lab.eng.brq.redhat.com
>>>> Server vm-127.idm.lab.eng.brq.redhat.com can't contact servers:
>>>> vm-244.idm.lab.eng.brq.redhat.com,
>>>> vm-056.idm.lab.eng.brq.redhat.com, vm-036.idm.lab.eng.brq.redhat.com
>>>> Server vm-244.idm.lab.eng.brq.redhat.com can't contact servers:
>>>> vm-056.idm.lab.eng.brq.redhat.com, vm-127.idm.lab.eng.brq.redhat.com
>>>> Continue to delete? [no]: yes
>>>> Waiting for removal of replication agreements
>>>> unexpected error: limits exceeded for this query
>>>>
>>>> I would expect this operation to delete 4 replication agreements on
>>>> all nodes:
>>>> vm056 - vm175
>>>> vm127 - vm175
>>>> vm244 - vm175
>>>> vm036 - vm175
>>>>
>>>> However an arbitrary set of replication agreements was deleted on
>>>> each node leading to total infrastructure inconsistency:
>>>> ===============================================================
>>>> vm056**thought the topology was as follows:
>>>> vm056 vm036
>>>> / |
>>>> vm175 |
>>>> / \ |
>>>> vm127 vm244
>>>> [10:28:55]ofayans at vm-056:~]$ ipa topologysegment-find realm
>>>> ------------------
>>>> 4 segments matched
>>>> ------------------
>>>> Segment name: 036-to-244
>>>> Left node: vm-036.idm.lab.eng.brq.redhat.com
>>>> Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>> Connectivity: both
>>>>
>>>> Segment name:
>>>> vm-036.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
>>>> Left node: vm-036.idm.lab.eng.brq.redhat.com
>>>> Right node: vm-175.idm.lab.eng.brq.redhat.com
>>>> Connectivity: both
>>>>
>>>> Segment name:
>>>> vm-127.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
>>>> Left node: vm-127.idm.lab.eng.brq.redhat.com
>>>> Right node: vm-175.idm.lab.eng.brq.redhat.com
>>>> Connectivity: both
>>>>
>>>> Segment name:
>>>> vm-175.idm.lab.eng.brq.redhat.com-to-vm-244.idm.lab.eng.brq.redhat.com
>>>> Left node: vm-175.idm.lab.eng.brq.redhat.com
>>>> Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>> Connectivity: both
>>>> ----------------------------
>>>> Number of entries returned 4
>>>> ----------------------------
>>>> ===============================================================
>>>> both vm036**vm244 thought the topology was as follows:
>>>> vm056 vm036
>>>> \ |
>>>> vm175 |
>>>> / |
>>>> vm127 vm244
>>>>
>>>> [10:26:23]ofayans at vm-036:~]$ ipa topologysegment-find
>>>> Suffix name: realm
>>>> ------------------
>>>> 3 segments matched
>>>> ------------------
>>>> Segment name: 036-to-244
>>>> Left node: vm-036.idm.lab.eng.brq.redhat.com
>>>> Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>> Connectivity: both
>>>>
>>>> Segment name:
>>>> vm-056.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
>>>> Left node: vm-056.idm.lab.eng.brq.redhat.com
>>>> Right node: vm-175.idm.lab.eng.brq.redhat.com
>>>> Connectivity: both
>>>>
>>>> Segment name:
>>>> vm-127.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
>>>> Left node: vm-127.idm.lab.eng.brq.redhat.com
>>>> Right node: vm-175.idm.lab.eng.brq.redhat.com
>>>> Connectivity: both
>>>> ----------------------------
>>>> Number of entries returned 3
>>>> ----------------------------
>>>>
>>>> ===============================================================
>>>> **vm127 thought the topology was as follows:
>>>> vm056 vm036
>>>> \ / |
>>>> vm175 |
>>>> \ |
>>>> vm127 vm244
>>>>
>>>> [10:31:08]ofayans at vm-127:~]$ ipa topologysegment-find realm
>>>> ------------------
>>>> 4 segments matched
>>>> ------------------
>>>> Segment name: 036-to-244
>>>> Left node: vm-036.idm.lab.eng.brq.redhat.com
>>>> Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>> Connectivity: both
>>>>
>>>> Segment name:
>>>> vm-036.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
>>>> Left node: vm-036.idm.lab.eng.brq.redhat.com
>>>> Right node: vm-175.idm.lab.eng.brq.redhat.com
>>>> Connectivity: both
>>>>
>>>> Segment name:
>>>> vm-056.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
>>>> Left node: vm-056.idm.lab.eng.brq.redhat.com
>>>> Right node: vm-175.idm.lab.eng.brq.redhat.com
>>>> Connectivity: both
>>>>
>>>> Segment name:
>>>> vm-175.idm.lab.eng.brq.redhat.com-to-vm-244.idm.lab.eng.brq.redhat.com
>>>> Left node: vm-175.idm.lab.eng.brq.redhat.com
>>>> Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>> Connectivity: both
>>>> ----------------------------
>>>> Number of entries returned 4
>>>> ----------------------------
>>>>
>>>> If I, for example, add a segment connecting vm127 and vm244, these
>>>> two nodes will not synchronize the topology info:
>>>>
>>>> [10:51:03]ofayans at vm-127:~]$ ipa topologysegment-add realm
>>>> 127-to-244 --leftnode=vm-127.idm.lab.eng.brq.redhat.com
>>>> --rightnode=vm-244.idm.lab.eng.brq.redhat.com --direction=both
>>>> --------------------------
>>>> Added segment "127-to-244"
>>>> --------------------------
>>>> Segment name: 127-to-244
>>>> Left node: vm-127.idm.lab.eng.brq.redhat.com
>>>> Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>> Connectivity: both
>>>> [10:53:33]ofayans at vm-127:~]$ ipa topologysegment-find realm
>>>> ------------------
>>>> 5 segments matched
>>>> ------------------
>>>> Segment name: 036-to-244
>>>> Left node: vm-036.idm.lab.eng.brq.redhat.com
>>>> Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>> Connectivity: both
>>>>
>>>> Segment name: 127-to-244
>>>> Left node: vm-127.idm.lab.eng.brq.redhat.com
>>>> Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>> Connectivity: both
>>>>
>>>> Segment name:
>>>> vm-036.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
>>>> Left node: vm-036.idm.lab.eng.brq.redhat.com
>>>> Right node: vm-175.idm.lab.eng.brq.redhat.com
>>>> Connectivity: both
>>>>
>>>> Segment name:
>>>> vm-056.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
>>>> Left node: vm-056.idm.lab.eng.brq.redhat.com
>>>> Right node: vm-175.idm.lab.eng.brq.redhat.com
>>>> Connectivity: both
>>>>
>>>> Segment name:
>>>> vm-175.idm.lab.eng.brq.redhat.com-to-vm-244.idm.lab.eng.brq.redhat.com
>>>> Left node: vm-175.idm.lab.eng.brq.redhat.com
>>>> Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>> Connectivity: both
>>>> ----------------------------
>>>> Number of entries returned 5
>>>> ----------------------------
>>>> [10:54:02]ofayans at vm-127:~]$
>>>>
>>>> =============================================================
>>>>
>>>> [10:49:38]ofayans at vm-244:~]$ ipa topologysegment-find realm
>>>> ------------------
>>>> 3 segments matched
>>>> ------------------
>>>> Segment name: 036-to-244
>>>> Left node: vm-036.idm.lab.eng.brq.redhat.com
>>>> Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>> Connectivity: both
>>>>
>>>> Segment name: 127-to-244
>>>> Left node: vm-127.idm.lab.eng.brq.redhat.com
>>>> Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>> Connectivity: both
>>>>
>>>> Segment name:
>>>> vm-056.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
>>>> Left node: vm-056.idm.lab.eng.brq.redhat.com
>>>> Right node: vm-175.idm.lab.eng.brq.redhat.com
>>>> Connectivity: both
>>>> ----------------------------
>>>> Number of entries returned 3
>>>> ----------------------------
>>>> [10:56:34]ofayans at vm-244:~]$
>>>>
>>>> Conclusion:
>>>> We either should completely prohibit the removal of the middle
>>>> nodes (I mean, nodes that hide another active nodes),
>>>> or at the removal stage first recalculate the resulting topology
>>>> and send it to all nodes before actual removal.
>>>> --
>>>> Oleg Fayans
>>>> Quality Engineer
>>>> FreeIPA team
>>>> RedHat.
>>>>
>>>>
>>>
>>>
>>>
>>
>> --
>> Oleg Fayans
>> Quality Engineer
>> FreeIPA team
>> RedHat.
>>
>>
>
>
>
--
Oleg Fayans
Quality Engineer
FreeIPA team
RedHat.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/freeipa-devel/attachments/20150624/72799d6b/attachment.htm>
More information about the Freeipa-devel
mailing list