[Freeipa-devel] Topology: Central node removal in star topology
Oleg Fayans
ofayans at redhat.com
Wed Jun 24 10:50:47 UTC 2015
On 06/24/2015 12:28 PM, Ludwig Krispenz wrote:
>
> On 06/24/2015 12:02 PM, Oleg Fayans wrote:
>>
>>
>> On 06/24/2015 11:47 AM, Ludwig Krispenz wrote:
>>>
>>> On 06/24/2015 11:36 AM, Oleg Fayans wrote:
>>>>
>>>>
>>>> On 06/24/2015 11:25 AM, Ludwig Krispenz wrote:
>>>>> Oleg,
>>>>>
>>>>> the topology plugin relies on existing connection between servers
>>>>> which remain in a topolgy. If you remove a central node in your
>>>>> topology you are asking for trouble.
>>>>> With Petr's patch it warns you that your topology will be
>>>>> disconnected, and if you insist we cannot guarantee anything.
>>>> Agree. I just wanted to try edge cases to see how one can break the
>>>> system :)
>>>>> should we completely prohibit this ? I don't know, I think you
>>>>> could also enforce an uninstall of vm175 with probably the same
>>>>> result.
>>>>> what you mean be calculating the remaining topology and send it to
>>>>> the remaining servers does not work, it would require to send a
>>>>> removal of a segment, which would be rejected.
>>>>>
>>>>> The topology is broken, and I don't know how much we should invest
>>>>> in making this info consistent on all servers.
>>>>>
>>>>> More interesting would be if we can heal this later by adding new
>>>>> segments.
>>>> Yes, here comes the biggest question raised from this case:
>>>> obviously, when none of the nodes possess the correct topology
>>>> information (including the one which deleted the central node),
>>>> there is no way to fix it by adding segments connecting the nodes
>>>> that became disconnected.
>>> It shoul not need the full information, but it has to be able to
>>> reach one of the nodes to be connected. when the topology is broken,
>>> you loose to feature to be ably to apply a change on any node, eg in
>>> your case if you want to connect vm036 and vm056 an have removed
>>> vm175, you have to do it on vm056, vm036 or vm244. This should work,
>>> if not we have to fix it - unless we completely prevent
>>> disconnecting a topology
>> Well, this is exactly the problem here: all replicas should contain
>> precise copies of all the info: accounts, hosts, sudorules, etc,
>> including topology information. However, if in this case I manually
>> connect disconnected node at vm127 (or vm056, does not matter) it
>> results in topology information inconsistency across the infrastructure:
>> This would be the topology from the point of view of vm127:
> did you add teh connection on vm127 or on vm244 ? sorry, but in these
> situations to understand what's going on, it can matter.
> to me it looks like you did it on vm127, so its there, it got
> replicated to vm244, but replicationback does not work and so the
> deletion of teh segs to vm175, which should still be in the changelogs
> of 036 and 244, don#t get to 127. Do you have something in the error
> logs of 244 ?
Yes, I added the connection on vm127. vm244 does not have anything in
the ldap errors log corresponding to the replication with vm127. In
fact, I tried to create a user on vm244 to see if it will be replicated
to vm127, and the user creation failed with the following error message:
Operations error: Allocation of a new value for range cn=posix
ids,cn=distributed numeric assignment plugin,cn=plugins,cn=config
failed! Unable to proceed.
Is it because the master node was deleted?
The corresponding message in the error log is
[24/Jun/2015:12:44:18 +0200] dna-plugin - dna_pre_op: no more values
available!!
>
>>
>> vm056 vm036
>> \ / |
>> vm175 |
>> \ |
>> vm127 vm244
>>
>> And this - from the point of view of vm244 and vm036
>>
>> vm056 vm036
>> \ |
>> vm175 |
>> |
>> vm127 ----- vm244
>>>> I still think that the recalculation of the resulting tree should
>>>> be done at least on the node that performs the removal action. And
>>>> when later some other node gets connected, it should understand
>>>> somehow that it's topology information is outdated
>>>>>
>>>>> Ludwig
>>>>> On 06/24/2015 11:04 AM, Oleg Fayans wrote:
>>>>>> Hi everybody,
>>>>>>
>>>>>> Current implementation of topology plugin (including patch 878
>>>>>> from Petr) allows the deletion of the central node in the star
>>>>>> topology.
>>>>>> I had the following topology:
>>>>>>
>>>>>> vm056 vm036
>>>>>> \ / |
>>>>>> vm175 |
>>>>>> / \ |
>>>>>> vm127 vm244
>>>>>>
>>>>>> I was able to remove node vm175 from node vm244:
>>>>>>
>>>>>> [17:54:48]ofayans at vm-244:~]$ ipa-replica-manage del
>>>>>> vm-175.idm.lab.eng.brq.redhat.com
>>>>>> Topology after removal of vm-175.idm.lab.eng.brq.redhat.com will
>>>>>> be disconnected:
>>>>>> Server vm-036.idm.lab.eng.brq.redhat.com can't contact servers:
>>>>>> vm-056.idm.lab.eng.brq.redhat.com, vm-127.idm.lab.eng.brq.redhat.com
>>>>>> Server vm-056.idm.lab.eng.brq.redhat.com can't contact servers:
>>>>>> vm-244.idm.lab.eng.brq.redhat.com,
>>>>>> vm-036.idm.lab.eng.brq.redhat.com, vm-127.idm.lab.eng.brq.redhat.com
>>>>>> Server vm-127.idm.lab.eng.brq.redhat.com can't contact servers:
>>>>>> vm-244.idm.lab.eng.brq.redhat.com,
>>>>>> vm-056.idm.lab.eng.brq.redhat.com, vm-036.idm.lab.eng.brq.redhat.com
>>>>>> Server vm-244.idm.lab.eng.brq.redhat.com can't contact servers:
>>>>>> vm-056.idm.lab.eng.brq.redhat.com, vm-127.idm.lab.eng.brq.redhat.com
>>>>>> Continue to delete? [no]: yes
>>>>>> Waiting for removal of replication agreements
>>>>>> unexpected error: limits exceeded for this query
>>>>>>
>>>>>> I would expect this operation to delete 4 replication agreements
>>>>>> on all nodes:
>>>>>> vm056 - vm175
>>>>>> vm127 - vm175
>>>>>> vm244 - vm175
>>>>>> vm036 - vm175
>>>>>>
>>>>>> However an arbitrary set of replication agreements was deleted on
>>>>>> each node leading to total infrastructure inconsistency:
>>>>>> ===============================================================
>>>>>> vm056**thought the topology was as follows:
>>>>>> vm056 vm036
>>>>>> / |
>>>>>> vm175 |
>>>>>> / \ |
>>>>>> vm127 vm244
>>>>>> [10:28:55]ofayans at vm-056:~]$ ipa topologysegment-find realm
>>>>>> ------------------
>>>>>> 4 segments matched
>>>>>> ------------------
>>>>>> Segment name: 036-to-244
>>>>>> Left node: vm-036.idm.lab.eng.brq.redhat.com
>>>>>> Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>>>> Connectivity: both
>>>>>>
>>>>>> Segment name:
>>>>>> vm-036.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
>>>>>> Left node: vm-036.idm.lab.eng.brq.redhat.com
>>>>>> Right node: vm-175.idm.lab.eng.brq.redhat.com
>>>>>> Connectivity: both
>>>>>>
>>>>>> Segment name:
>>>>>> vm-127.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
>>>>>> Left node: vm-127.idm.lab.eng.brq.redhat.com
>>>>>> Right node: vm-175.idm.lab.eng.brq.redhat.com
>>>>>> Connectivity: both
>>>>>>
>>>>>> Segment name:
>>>>>> vm-175.idm.lab.eng.brq.redhat.com-to-vm-244.idm.lab.eng.brq.redhat.com
>>>>>> Left node: vm-175.idm.lab.eng.brq.redhat.com
>>>>>> Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>>>> Connectivity: both
>>>>>> ----------------------------
>>>>>> Number of entries returned 4
>>>>>> ----------------------------
>>>>>> ===============================================================
>>>>>> both vm036**vm244 thought the topology was as follows:
>>>>>> vm056 vm036
>>>>>> \ |
>>>>>> vm175 |
>>>>>> / |
>>>>>> vm127 vm244
>>>>>>
>>>>>> [10:26:23]ofayans at vm-036:~]$ ipa topologysegment-find
>>>>>> Suffix name: realm
>>>>>> ------------------
>>>>>> 3 segments matched
>>>>>> ------------------
>>>>>> Segment name: 036-to-244
>>>>>> Left node: vm-036.idm.lab.eng.brq.redhat.com
>>>>>> Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>>>> Connectivity: both
>>>>>>
>>>>>> Segment name:
>>>>>> vm-056.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
>>>>>> Left node: vm-056.idm.lab.eng.brq.redhat.com
>>>>>> Right node: vm-175.idm.lab.eng.brq.redhat.com
>>>>>> Connectivity: both
>>>>>>
>>>>>> Segment name:
>>>>>> vm-127.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
>>>>>> Left node: vm-127.idm.lab.eng.brq.redhat.com
>>>>>> Right node: vm-175.idm.lab.eng.brq.redhat.com
>>>>>> Connectivity: both
>>>>>> ----------------------------
>>>>>> Number of entries returned 3
>>>>>> ----------------------------
>>>>>>
>>>>>> ===============================================================
>>>>>> **vm127 thought the topology was as follows:
>>>>>> vm056 vm036
>>>>>> \ / |
>>>>>> vm175 |
>>>>>> \ |
>>>>>> vm127 vm244
>>>>>>
>>>>>> [10:31:08]ofayans at vm-127:~]$ ipa topologysegment-find realm
>>>>>> ------------------
>>>>>> 4 segments matched
>>>>>> ------------------
>>>>>> Segment name: 036-to-244
>>>>>> Left node: vm-036.idm.lab.eng.brq.redhat.com
>>>>>> Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>>>> Connectivity: both
>>>>>>
>>>>>> Segment name:
>>>>>> vm-036.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
>>>>>> Left node: vm-036.idm.lab.eng.brq.redhat.com
>>>>>> Right node: vm-175.idm.lab.eng.brq.redhat.com
>>>>>> Connectivity: both
>>>>>>
>>>>>> Segment name:
>>>>>> vm-056.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
>>>>>> Left node: vm-056.idm.lab.eng.brq.redhat.com
>>>>>> Right node: vm-175.idm.lab.eng.brq.redhat.com
>>>>>> Connectivity: both
>>>>>>
>>>>>> Segment name:
>>>>>> vm-175.idm.lab.eng.brq.redhat.com-to-vm-244.idm.lab.eng.brq.redhat.com
>>>>>> Left node: vm-175.idm.lab.eng.brq.redhat.com
>>>>>> Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>>>> Connectivity: both
>>>>>> ----------------------------
>>>>>> Number of entries returned 4
>>>>>> ----------------------------
>>>>>>
>>>>>> If I, for example, add a segment connecting vm127 and vm244,
>>>>>> these two nodes will not synchronize the topology info:
>>>>>>
>>>>>> [10:51:03]ofayans at vm-127:~]$ ipa topologysegment-add realm
>>>>>> 127-to-244 --leftnode=vm-127.idm.lab.eng.brq.redhat.com
>>>>>> --rightnode=vm-244.idm.lab.eng.brq.redhat.com --direction=both
>>>>>> --------------------------
>>>>>> Added segment "127-to-244"
>>>>>> --------------------------
>>>>>> Segment name: 127-to-244
>>>>>> Left node: vm-127.idm.lab.eng.brq.redhat.com
>>>>>> Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>>>> Connectivity: both
>>>>>> [10:53:33]ofayans at vm-127:~]$ ipa topologysegment-find realm
>>>>>> ------------------
>>>>>> 5 segments matched
>>>>>> ------------------
>>>>>> Segment name: 036-to-244
>>>>>> Left node: vm-036.idm.lab.eng.brq.redhat.com
>>>>>> Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>>>> Connectivity: both
>>>>>>
>>>>>> Segment name: 127-to-244
>>>>>> Left node: vm-127.idm.lab.eng.brq.redhat.com
>>>>>> Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>>>> Connectivity: both
>>>>>>
>>>>>> Segment name:
>>>>>> vm-036.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
>>>>>> Left node: vm-036.idm.lab.eng.brq.redhat.com
>>>>>> Right node: vm-175.idm.lab.eng.brq.redhat.com
>>>>>> Connectivity: both
>>>>>>
>>>>>> Segment name:
>>>>>> vm-056.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
>>>>>> Left node: vm-056.idm.lab.eng.brq.redhat.com
>>>>>> Right node: vm-175.idm.lab.eng.brq.redhat.com
>>>>>> Connectivity: both
>>>>>>
>>>>>> Segment name:
>>>>>> vm-175.idm.lab.eng.brq.redhat.com-to-vm-244.idm.lab.eng.brq.redhat.com
>>>>>> Left node: vm-175.idm.lab.eng.brq.redhat.com
>>>>>> Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>>>> Connectivity: both
>>>>>> ----------------------------
>>>>>> Number of entries returned 5
>>>>>> ----------------------------
>>>>>> [10:54:02]ofayans at vm-127:~]$
>>>>>>
>>>>>> =============================================================
>>>>>>
>>>>>> [10:49:38]ofayans at vm-244:~]$ ipa topologysegment-find realm
>>>>>> ------------------
>>>>>> 3 segments matched
>>>>>> ------------------
>>>>>> Segment name: 036-to-244
>>>>>> Left node: vm-036.idm.lab.eng.brq.redhat.com
>>>>>> Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>>>> Connectivity: both
>>>>>>
>>>>>> Segment name: 127-to-244
>>>>>> Left node: vm-127.idm.lab.eng.brq.redhat.com
>>>>>> Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>>>> Connectivity: both
>>>>>>
>>>>>> Segment name:
>>>>>> vm-056.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
>>>>>> Left node: vm-056.idm.lab.eng.brq.redhat.com
>>>>>> Right node: vm-175.idm.lab.eng.brq.redhat.com
>>>>>> Connectivity: both
>>>>>> ----------------------------
>>>>>> Number of entries returned 3
>>>>>> ----------------------------
>>>>>> [10:56:34]ofayans at vm-244:~]$
>>>>>>
>>>>>> Conclusion:
>>>>>> We either should completely prohibit the removal of the middle
>>>>>> nodes (I mean, nodes that hide another active nodes),
>>>>>> or at the removal stage first recalculate the resulting topology
>>>>>> and send it to all nodes before actual removal.
>>>>>> --
>>>>>> Oleg Fayans
>>>>>> Quality Engineer
>>>>>> FreeIPA team
>>>>>> RedHat.
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> Oleg Fayans
>>>> Quality Engineer
>>>> FreeIPA team
>>>> RedHat.
>>>>
>>>>
>>>
>>>
>>>
>>
>> --
>> Oleg Fayans
>> Quality Engineer
>> FreeIPA team
>> RedHat.
>>
>>
>
>
>
--
Oleg Fayans
Quality Engineer
FreeIPA team
RedHat.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/freeipa-devel/attachments/20150624/676ca9b9/attachment.htm>
More information about the Freeipa-devel
mailing list