[Freeipa-devel] Topology: Central node removal in star topology
Oleg Fayans
ofayans at redhat.com
Wed Jun 24 10:03:50 UTC 2015
On 06/24/2015 12:02 PM, Oleg Fayans wrote:
>
>
> On 06/24/2015 11:47 AM, Ludwig Krispenz wrote:
>>
>> On 06/24/2015 11:36 AM, Oleg Fayans wrote:
>>>
>>>
>>> On 06/24/2015 11:25 AM, Ludwig Krispenz wrote:
>>>> Oleg,
>>>>
>>>> the topology plugin relies on existing connection between servers
>>>> which remain in a topolgy. If you remove a central node in your
>>>> topology you are asking for trouble.
>>>> With Petr's patch it warns you that your topology will be
>>>> disconnected, and if you insist we cannot guarantee anything.
>>> Agree. I just wanted to try edge cases to see how one can break the
>>> system :)
>>>> should we completely prohibit this ? I don't know, I think you
>>>> could also enforce an uninstall of vm175 with probably the same result.
>>>> what you mean be calculating the remaining topology and send it to
>>>> the remaining servers does not work, it would require to send a
>>>> removal of a segment, which would be rejected.
>>>>
>>>> The topology is broken, and I don't know how much we should invest
>>>> in making this info consistent on all servers.
>>>>
>>>> More interesting would be if we can heal this later by adding new
>>>> segments.
>>> Yes, here comes the biggest question raised from this case:
>>> obviously, when none of the nodes possess the correct topology
>>> information (including the one which deleted the central node),
>>> there is no way to fix it by adding segments connecting the nodes
>>> that became disconnected.
>> It shoul not need the full information, but it has to be able to
>> reach one of the nodes to be connected. when the topology is broken,
>> you loose to feature to be ably to apply a change on any node, eg in
>> your case if you want to connect vm036 and vm056 an have removed
>> vm175, you have to do it on vm056, vm036 or vm244. This should work,
>> if not we have to fix it - unless we completely prevent disconnecting
>> a topology
> Well, this is exactly the problem here: all replicas should contain
> precise copies of all the info: accounts, hosts, sudorules, etc,
> including topology information. However, if in this case I manually
> connect disconnected node at vm127 (or vm056, does not matter) it
> results in topology information inconsistency across the infrastructure:
> This would be the topology from the point of view of vm127:
>
> vm056 vm036
> \ / |
> vm175 |
> \ |
> vm127 vm244
sorry, I meant
vm056 vm036
\ / |
vm175 |
\ |
vm127 ----- vm244
>
> And this - from the point of view of vm244 and vm036
>
> vm056 vm036
> \ |
> vm175 |
> |
> vm127 ----- vm244
>>> I still think that the recalculation of the resulting tree should be
>>> done at least on the node that performs the removal action. And when
>>> later some other node gets connected, it should understand somehow
>>> that it's topology information is outdated
>>>>
>>>> Ludwig
>>>> On 06/24/2015 11:04 AM, Oleg Fayans wrote:
>>>>> Hi everybody,
>>>>>
>>>>> Current implementation of topology plugin (including patch 878
>>>>> from Petr) allows the deletion of the central node in the star
>>>>> topology.
>>>>> I had the following topology:
>>>>>
>>>>> vm056 vm036
>>>>> \ / |
>>>>> vm175 |
>>>>> / \ |
>>>>> vm127 vm244
>>>>>
>>>>> I was able to remove node vm175 from node vm244:
>>>>>
>>>>> [17:54:48]ofayans at vm-244:~]$ ipa-replica-manage del
>>>>> vm-175.idm.lab.eng.brq.redhat.com
>>>>> Topology after removal of vm-175.idm.lab.eng.brq.redhat.com will
>>>>> be disconnected:
>>>>> Server vm-036.idm.lab.eng.brq.redhat.com can't contact servers:
>>>>> vm-056.idm.lab.eng.brq.redhat.com, vm-127.idm.lab.eng.brq.redhat.com
>>>>> Server vm-056.idm.lab.eng.brq.redhat.com can't contact servers:
>>>>> vm-244.idm.lab.eng.brq.redhat.com,
>>>>> vm-036.idm.lab.eng.brq.redhat.com, vm-127.idm.lab.eng.brq.redhat.com
>>>>> Server vm-127.idm.lab.eng.brq.redhat.com can't contact servers:
>>>>> vm-244.idm.lab.eng.brq.redhat.com,
>>>>> vm-056.idm.lab.eng.brq.redhat.com, vm-036.idm.lab.eng.brq.redhat.com
>>>>> Server vm-244.idm.lab.eng.brq.redhat.com can't contact servers:
>>>>> vm-056.idm.lab.eng.brq.redhat.com, vm-127.idm.lab.eng.brq.redhat.com
>>>>> Continue to delete? [no]: yes
>>>>> Waiting for removal of replication agreements
>>>>> unexpected error: limits exceeded for this query
>>>>>
>>>>> I would expect this operation to delete 4 replication agreements
>>>>> on all nodes:
>>>>> vm056 - vm175
>>>>> vm127 - vm175
>>>>> vm244 - vm175
>>>>> vm036 - vm175
>>>>>
>>>>> However an arbitrary set of replication agreements was deleted on
>>>>> each node leading to total infrastructure inconsistency:
>>>>> ===============================================================
>>>>> vm056**thought the topology was as follows:
>>>>> vm056 vm036
>>>>> / |
>>>>> vm175 |
>>>>> / \ |
>>>>> vm127 vm244
>>>>> [10:28:55]ofayans at vm-056:~]$ ipa topologysegment-find realm
>>>>> ------------------
>>>>> 4 segments matched
>>>>> ------------------
>>>>> Segment name: 036-to-244
>>>>> Left node: vm-036.idm.lab.eng.brq.redhat.com
>>>>> Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>>> Connectivity: both
>>>>>
>>>>> Segment name:
>>>>> vm-036.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
>>>>> Left node: vm-036.idm.lab.eng.brq.redhat.com
>>>>> Right node: vm-175.idm.lab.eng.brq.redhat.com
>>>>> Connectivity: both
>>>>>
>>>>> Segment name:
>>>>> vm-127.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
>>>>> Left node: vm-127.idm.lab.eng.brq.redhat.com
>>>>> Right node: vm-175.idm.lab.eng.brq.redhat.com
>>>>> Connectivity: both
>>>>>
>>>>> Segment name:
>>>>> vm-175.idm.lab.eng.brq.redhat.com-to-vm-244.idm.lab.eng.brq.redhat.com
>>>>> Left node: vm-175.idm.lab.eng.brq.redhat.com
>>>>> Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>>> Connectivity: both
>>>>> ----------------------------
>>>>> Number of entries returned 4
>>>>> ----------------------------
>>>>> ===============================================================
>>>>> both vm036**vm244 thought the topology was as follows:
>>>>> vm056 vm036
>>>>> \ |
>>>>> vm175 |
>>>>> / |
>>>>> vm127 vm244
>>>>>
>>>>> [10:26:23]ofayans at vm-036:~]$ ipa topologysegment-find
>>>>> Suffix name: realm
>>>>> ------------------
>>>>> 3 segments matched
>>>>> ------------------
>>>>> Segment name: 036-to-244
>>>>> Left node: vm-036.idm.lab.eng.brq.redhat.com
>>>>> Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>>> Connectivity: both
>>>>>
>>>>> Segment name:
>>>>> vm-056.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
>>>>> Left node: vm-056.idm.lab.eng.brq.redhat.com
>>>>> Right node: vm-175.idm.lab.eng.brq.redhat.com
>>>>> Connectivity: both
>>>>>
>>>>> Segment name:
>>>>> vm-127.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
>>>>> Left node: vm-127.idm.lab.eng.brq.redhat.com
>>>>> Right node: vm-175.idm.lab.eng.brq.redhat.com
>>>>> Connectivity: both
>>>>> ----------------------------
>>>>> Number of entries returned 3
>>>>> ----------------------------
>>>>>
>>>>> ===============================================================
>>>>> **vm127 thought the topology was as follows:
>>>>> vm056 vm036
>>>>> \ / |
>>>>> vm175 |
>>>>> \ |
>>>>> vm127 vm244
>>>>>
>>>>> [10:31:08]ofayans at vm-127:~]$ ipa topologysegment-find realm
>>>>> ------------------
>>>>> 4 segments matched
>>>>> ------------------
>>>>> Segment name: 036-to-244
>>>>> Left node: vm-036.idm.lab.eng.brq.redhat.com
>>>>> Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>>> Connectivity: both
>>>>>
>>>>> Segment name:
>>>>> vm-036.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
>>>>> Left node: vm-036.idm.lab.eng.brq.redhat.com
>>>>> Right node: vm-175.idm.lab.eng.brq.redhat.com
>>>>> Connectivity: both
>>>>>
>>>>> Segment name:
>>>>> vm-056.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
>>>>> Left node: vm-056.idm.lab.eng.brq.redhat.com
>>>>> Right node: vm-175.idm.lab.eng.brq.redhat.com
>>>>> Connectivity: both
>>>>>
>>>>> Segment name:
>>>>> vm-175.idm.lab.eng.brq.redhat.com-to-vm-244.idm.lab.eng.brq.redhat.com
>>>>> Left node: vm-175.idm.lab.eng.brq.redhat.com
>>>>> Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>>> Connectivity: both
>>>>> ----------------------------
>>>>> Number of entries returned 4
>>>>> ----------------------------
>>>>>
>>>>> If I, for example, add a segment connecting vm127 and vm244, these
>>>>> two nodes will not synchronize the topology info:
>>>>>
>>>>> [10:51:03]ofayans at vm-127:~]$ ipa topologysegment-add realm
>>>>> 127-to-244 --leftnode=vm-127.idm.lab.eng.brq.redhat.com
>>>>> --rightnode=vm-244.idm.lab.eng.brq.redhat.com --direction=both
>>>>> --------------------------
>>>>> Added segment "127-to-244"
>>>>> --------------------------
>>>>> Segment name: 127-to-244
>>>>> Left node: vm-127.idm.lab.eng.brq.redhat.com
>>>>> Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>>> Connectivity: both
>>>>> [10:53:33]ofayans at vm-127:~]$ ipa topologysegment-find realm
>>>>> ------------------
>>>>> 5 segments matched
>>>>> ------------------
>>>>> Segment name: 036-to-244
>>>>> Left node: vm-036.idm.lab.eng.brq.redhat.com
>>>>> Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>>> Connectivity: both
>>>>>
>>>>> Segment name: 127-to-244
>>>>> Left node: vm-127.idm.lab.eng.brq.redhat.com
>>>>> Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>>> Connectivity: both
>>>>>
>>>>> Segment name:
>>>>> vm-036.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
>>>>> Left node: vm-036.idm.lab.eng.brq.redhat.com
>>>>> Right node: vm-175.idm.lab.eng.brq.redhat.com
>>>>> Connectivity: both
>>>>>
>>>>> Segment name:
>>>>> vm-056.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
>>>>> Left node: vm-056.idm.lab.eng.brq.redhat.com
>>>>> Right node: vm-175.idm.lab.eng.brq.redhat.com
>>>>> Connectivity: both
>>>>>
>>>>> Segment name:
>>>>> vm-175.idm.lab.eng.brq.redhat.com-to-vm-244.idm.lab.eng.brq.redhat.com
>>>>> Left node: vm-175.idm.lab.eng.brq.redhat.com
>>>>> Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>>> Connectivity: both
>>>>> ----------------------------
>>>>> Number of entries returned 5
>>>>> ----------------------------
>>>>> [10:54:02]ofayans at vm-127:~]$
>>>>>
>>>>> =============================================================
>>>>>
>>>>> [10:49:38]ofayans at vm-244:~]$ ipa topologysegment-find realm
>>>>> ------------------
>>>>> 3 segments matched
>>>>> ------------------
>>>>> Segment name: 036-to-244
>>>>> Left node: vm-036.idm.lab.eng.brq.redhat.com
>>>>> Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>>> Connectivity: both
>>>>>
>>>>> Segment name: 127-to-244
>>>>> Left node: vm-127.idm.lab.eng.brq.redhat.com
>>>>> Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>>> Connectivity: both
>>>>>
>>>>> Segment name:
>>>>> vm-056.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
>>>>> Left node: vm-056.idm.lab.eng.brq.redhat.com
>>>>> Right node: vm-175.idm.lab.eng.brq.redhat.com
>>>>> Connectivity: both
>>>>> ----------------------------
>>>>> Number of entries returned 3
>>>>> ----------------------------
>>>>> [10:56:34]ofayans at vm-244:~]$
>>>>>
>>>>> Conclusion:
>>>>> We either should completely prohibit the removal of the middle
>>>>> nodes (I mean, nodes that hide another active nodes),
>>>>> or at the removal stage first recalculate the resulting topology
>>>>> and send it to all nodes before actual removal.
>>>>> --
>>>>> Oleg Fayans
>>>>> Quality Engineer
>>>>> FreeIPA team
>>>>> RedHat.
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>> --
>>> Oleg Fayans
>>> Quality Engineer
>>> FreeIPA team
>>> RedHat.
>>>
>>>
>>
>>
>>
>
> --
> Oleg Fayans
> Quality Engineer
> FreeIPA team
> RedHat.
>
>
--
Oleg Fayans
Quality Engineer
FreeIPA team
RedHat.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/freeipa-devel/attachments/20150624/880b96d2/attachment.htm>
More information about the Freeipa-devel
mailing list