[Freeipa-devel] Topology: Central node removal in star topology
Ludwig Krispenz
lkrispen at redhat.com
Wed Jun 24 11:09:46 UTC 2015
On 06/24/2015 12:50 PM, Oleg Fayans wrote:
>
>
> On 06/24/2015 12:28 PM, Ludwig Krispenz wrote:
>>
>> On 06/24/2015 12:02 PM, Oleg Fayans wrote:
>>>
>>>
>>> On 06/24/2015 11:47 AM, Ludwig Krispenz wrote:
>>>>
>>>> On 06/24/2015 11:36 AM, Oleg Fayans wrote:
>>>>>
>>>>>
>>>>> On 06/24/2015 11:25 AM, Ludwig Krispenz wrote:
>>>>>> Oleg,
>>>>>>
>>>>>> the topology plugin relies on existing connection between servers
>>>>>> which remain in a topolgy. If you remove a central node in your
>>>>>> topology you are asking for trouble.
>>>>>> With Petr's patch it warns you that your topology will be
>>>>>> disconnected, and if you insist we cannot guarantee anything.
>>>>> Agree. I just wanted to try edge cases to see how one can break
>>>>> the system :)
>>>>>> should we completely prohibit this ? I don't know, I think you
>>>>>> could also enforce an uninstall of vm175 with probably the same
>>>>>> result.
>>>>>> what you mean be calculating the remaining topology and send it
>>>>>> to the remaining servers does not work, it would require to send
>>>>>> a removal of a segment, which would be rejected.
>>>>>>
>>>>>> The topology is broken, and I don't know how much we should
>>>>>> invest in making this info consistent on all servers.
>>>>>>
>>>>>> More interesting would be if we can heal this later by adding new
>>>>>> segments.
>>>>> Yes, here comes the biggest question raised from this case:
>>>>> obviously, when none of the nodes possess the correct topology
>>>>> information (including the one which deleted the central node),
>>>>> there is no way to fix it by adding segments connecting the nodes
>>>>> that became disconnected.
>>>> It shoul not need the full information, but it has to be able to
>>>> reach one of the nodes to be connected. when the topology is
>>>> broken, you loose to feature to be ably to apply a change on any
>>>> node, eg in your case if you want to connect vm036 and vm056 an
>>>> have removed vm175, you have to do it on vm056, vm036 or vm244.
>>>> This should work, if not we have to fix it - unless we completely
>>>> prevent disconnecting a topology
>>> Well, this is exactly the problem here: all replicas should contain
>>> precise copies of all the info: accounts, hosts, sudorules, etc,
>>> including topology information. However, if in this case I manually
>>> connect disconnected node at vm127 (or vm056, does not matter) it
>>> results in topology information inconsistency across the infrastructure:
>>> This would be the topology from the point of view of vm127:
>> did you add teh connection on vm127 or on vm244 ? sorry, but in these
>> situations to understand what's going on, it can matter.
>> to me it looks like you did it on vm127, so its there, it got
>> replicated to vm244, but replicationback does not work and so the
>> deletion of teh segs to vm175, which should still be in the
>> changelogs of 036 and 244, don#t get to 127. Do you have something in
>> the error logs of 244 ?
> Yes, I added the connection on vm127. vm244 does not have anything in
> the ldap errors log corresponding to the replication with vm127. In
> fact, I tried to create a user on vm244 to see if it will be
> replicated to vm127, and the user creation failed with the following
> error message:
> Operations error: Allocation of a new value for range cn=posix
> ids,cn=distributed numeric assignment plugin,cn=plugins,cn=config
> failed! Unable to proceed.
>
> Is it because the master node was deleted?
think so, yes.
There are probably more things to check before removing a server :-(
> The corresponding message in the error log is
> [24/Jun/2015:12:44:18 +0200] dna-plugin - dna_pre_op: no more values
> available!!
>>
>>>
>>> vm056 vm036
>>> \ / |
>>> vm175 |
>>> \ |
>>> vm127 vm244
>>>
>>> And this - from the point of view of vm244 and vm036
>>>
>>> vm056 vm036
>>> \ |
>>> vm175 |
>>> |
>>> vm127 ----- vm244
>>>>> I still think that the recalculation of the resulting tree should
>>>>> be done at least on the node that performs the removal action. And
>>>>> when later some other node gets connected, it should understand
>>>>> somehow that it's topology information is outdated
>>>>>>
>>>>>> Ludwig
>>>>>> On 06/24/2015 11:04 AM, Oleg Fayans wrote:
>>>>>>> Hi everybody,
>>>>>>>
>>>>>>> Current implementation of topology plugin (including patch 878
>>>>>>> from Petr) allows the deletion of the central node in the star
>>>>>>> topology.
>>>>>>> I had the following topology:
>>>>>>>
>>>>>>> vm056 vm036
>>>>>>> \ / |
>>>>>>> vm175 |
>>>>>>> / \ |
>>>>>>> vm127 vm244
>>>>>>>
>>>>>>> I was able to remove node vm175 from node vm244:
>>>>>>>
>>>>>>> [17:54:48]ofayans at vm-244:~]$ ipa-replica-manage del
>>>>>>> vm-175.idm.lab.eng.brq.redhat.com
>>>>>>> Topology after removal of vm-175.idm.lab.eng.brq.redhat.com will
>>>>>>> be disconnected:
>>>>>>> Server vm-036.idm.lab.eng.brq.redhat.com can't contact servers:
>>>>>>> vm-056.idm.lab.eng.brq.redhat.com, vm-127.idm.lab.eng.brq.redhat.com
>>>>>>> Server vm-056.idm.lab.eng.brq.redhat.com can't contact servers:
>>>>>>> vm-244.idm.lab.eng.brq.redhat.com,
>>>>>>> vm-036.idm.lab.eng.brq.redhat.com, vm-127.idm.lab.eng.brq.redhat.com
>>>>>>> Server vm-127.idm.lab.eng.brq.redhat.com can't contact servers:
>>>>>>> vm-244.idm.lab.eng.brq.redhat.com,
>>>>>>> vm-056.idm.lab.eng.brq.redhat.com, vm-036.idm.lab.eng.brq.redhat.com
>>>>>>> Server vm-244.idm.lab.eng.brq.redhat.com can't contact servers:
>>>>>>> vm-056.idm.lab.eng.brq.redhat.com, vm-127.idm.lab.eng.brq.redhat.com
>>>>>>> Continue to delete? [no]: yes
>>>>>>> Waiting for removal of replication agreements
>>>>>>> unexpected error: limits exceeded for this query
>>>>>>>
>>>>>>> I would expect this operation to delete 4 replication agreements
>>>>>>> on all nodes:
>>>>>>> vm056 - vm175
>>>>>>> vm127 - vm175
>>>>>>> vm244 - vm175
>>>>>>> vm036 - vm175
>>>>>>>
>>>>>>> However an arbitrary set of replication agreements was deleted
>>>>>>> on each node leading to total infrastructure inconsistency:
>>>>>>> ===============================================================
>>>>>>> vm056**thought the topology was as follows:
>>>>>>> vm056 vm036
>>>>>>> / |
>>>>>>> vm175 |
>>>>>>> / \ |
>>>>>>> vm127 vm244
>>>>>>> [10:28:55]ofayans at vm-056:~]$ ipa topologysegment-find realm
>>>>>>> ------------------
>>>>>>> 4 segments matched
>>>>>>> ------------------
>>>>>>> Segment name: 036-to-244
>>>>>>> Left node: vm-036.idm.lab.eng.brq.redhat.com
>>>>>>> Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>>>>> Connectivity: both
>>>>>>>
>>>>>>> Segment name:
>>>>>>> vm-036.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
>>>>>>> Left node: vm-036.idm.lab.eng.brq.redhat.com
>>>>>>> Right node: vm-175.idm.lab.eng.brq.redhat.com
>>>>>>> Connectivity: both
>>>>>>>
>>>>>>> Segment name:
>>>>>>> vm-127.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
>>>>>>> Left node: vm-127.idm.lab.eng.brq.redhat.com
>>>>>>> Right node: vm-175.idm.lab.eng.brq.redhat.com
>>>>>>> Connectivity: both
>>>>>>>
>>>>>>> Segment name:
>>>>>>> vm-175.idm.lab.eng.brq.redhat.com-to-vm-244.idm.lab.eng.brq.redhat.com
>>>>>>> Left node: vm-175.idm.lab.eng.brq.redhat.com
>>>>>>> Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>>>>> Connectivity: both
>>>>>>> ----------------------------
>>>>>>> Number of entries returned 4
>>>>>>> ----------------------------
>>>>>>> ===============================================================
>>>>>>> both vm036**vm244 thought the topology was as follows:
>>>>>>> vm056 vm036
>>>>>>> \ |
>>>>>>> vm175 |
>>>>>>> / |
>>>>>>> vm127 vm244
>>>>>>>
>>>>>>> [10:26:23]ofayans at vm-036:~]$ ipa topologysegment-find
>>>>>>> Suffix name: realm
>>>>>>> ------------------
>>>>>>> 3 segments matched
>>>>>>> ------------------
>>>>>>> Segment name: 036-to-244
>>>>>>> Left node: vm-036.idm.lab.eng.brq.redhat.com
>>>>>>> Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>>>>> Connectivity: both
>>>>>>>
>>>>>>> Segment name:
>>>>>>> vm-056.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
>>>>>>> Left node: vm-056.idm.lab.eng.brq.redhat.com
>>>>>>> Right node: vm-175.idm.lab.eng.brq.redhat.com
>>>>>>> Connectivity: both
>>>>>>>
>>>>>>> Segment name:
>>>>>>> vm-127.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
>>>>>>> Left node: vm-127.idm.lab.eng.brq.redhat.com
>>>>>>> Right node: vm-175.idm.lab.eng.brq.redhat.com
>>>>>>> Connectivity: both
>>>>>>> ----------------------------
>>>>>>> Number of entries returned 3
>>>>>>> ----------------------------
>>>>>>>
>>>>>>> ===============================================================
>>>>>>> **vm127 thought the topology was as follows:
>>>>>>> vm056 vm036
>>>>>>> \ / |
>>>>>>> vm175 |
>>>>>>> \ |
>>>>>>> vm127 vm244
>>>>>>>
>>>>>>> [10:31:08]ofayans at vm-127:~]$ ipa topologysegment-find realm
>>>>>>> ------------------
>>>>>>> 4 segments matched
>>>>>>> ------------------
>>>>>>> Segment name: 036-to-244
>>>>>>> Left node: vm-036.idm.lab.eng.brq.redhat.com
>>>>>>> Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>>>>> Connectivity: both
>>>>>>>
>>>>>>> Segment name:
>>>>>>> vm-036.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
>>>>>>> Left node: vm-036.idm.lab.eng.brq.redhat.com
>>>>>>> Right node: vm-175.idm.lab.eng.brq.redhat.com
>>>>>>> Connectivity: both
>>>>>>>
>>>>>>> Segment name:
>>>>>>> vm-056.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
>>>>>>> Left node: vm-056.idm.lab.eng.brq.redhat.com
>>>>>>> Right node: vm-175.idm.lab.eng.brq.redhat.com
>>>>>>> Connectivity: both
>>>>>>>
>>>>>>> Segment name:
>>>>>>> vm-175.idm.lab.eng.brq.redhat.com-to-vm-244.idm.lab.eng.brq.redhat.com
>>>>>>> Left node: vm-175.idm.lab.eng.brq.redhat.com
>>>>>>> Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>>>>> Connectivity: both
>>>>>>> ----------------------------
>>>>>>> Number of entries returned 4
>>>>>>> ----------------------------
>>>>>>>
>>>>>>> If I, for example, add a segment connecting vm127 and vm244,
>>>>>>> these two nodes will not synchronize the topology info:
>>>>>>>
>>>>>>> [10:51:03]ofayans at vm-127:~]$ ipa topologysegment-add realm
>>>>>>> 127-to-244 --leftnode=vm-127.idm.lab.eng.brq.redhat.com
>>>>>>> --rightnode=vm-244.idm.lab.eng.brq.redhat.com --direction=both
>>>>>>> --------------------------
>>>>>>> Added segment "127-to-244"
>>>>>>> --------------------------
>>>>>>> Segment name: 127-to-244
>>>>>>> Left node: vm-127.idm.lab.eng.brq.redhat.com
>>>>>>> Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>>>>> Connectivity: both
>>>>>>> [10:53:33]ofayans at vm-127:~]$ ipa topologysegment-find realm
>>>>>>> ------------------
>>>>>>> 5 segments matched
>>>>>>> ------------------
>>>>>>> Segment name: 036-to-244
>>>>>>> Left node: vm-036.idm.lab.eng.brq.redhat.com
>>>>>>> Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>>>>> Connectivity: both
>>>>>>>
>>>>>>> Segment name: 127-to-244
>>>>>>> Left node: vm-127.idm.lab.eng.brq.redhat.com
>>>>>>> Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>>>>> Connectivity: both
>>>>>>>
>>>>>>> Segment name:
>>>>>>> vm-036.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
>>>>>>> Left node: vm-036.idm.lab.eng.brq.redhat.com
>>>>>>> Right node: vm-175.idm.lab.eng.brq.redhat.com
>>>>>>> Connectivity: both
>>>>>>>
>>>>>>> Segment name:
>>>>>>> vm-056.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
>>>>>>> Left node: vm-056.idm.lab.eng.brq.redhat.com
>>>>>>> Right node: vm-175.idm.lab.eng.brq.redhat.com
>>>>>>> Connectivity: both
>>>>>>>
>>>>>>> Segment name:
>>>>>>> vm-175.idm.lab.eng.brq.redhat.com-to-vm-244.idm.lab.eng.brq.redhat.com
>>>>>>> Left node: vm-175.idm.lab.eng.brq.redhat.com
>>>>>>> Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>>>>> Connectivity: both
>>>>>>> ----------------------------
>>>>>>> Number of entries returned 5
>>>>>>> ----------------------------
>>>>>>> [10:54:02]ofayans at vm-127:~]$
>>>>>>>
>>>>>>> =============================================================
>>>>>>>
>>>>>>> [10:49:38]ofayans at vm-244:~]$ ipa topologysegment-find realm
>>>>>>> ------------------
>>>>>>> 3 segments matched
>>>>>>> ------------------
>>>>>>> Segment name: 036-to-244
>>>>>>> Left node: vm-036.idm.lab.eng.brq.redhat.com
>>>>>>> Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>>>>> Connectivity: both
>>>>>>>
>>>>>>> Segment name: 127-to-244
>>>>>>> Left node: vm-127.idm.lab.eng.brq.redhat.com
>>>>>>> Right node: vm-244.idm.lab.eng.brq.redhat.com
>>>>>>> Connectivity: both
>>>>>>>
>>>>>>> Segment name:
>>>>>>> vm-056.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
>>>>>>> Left node: vm-056.idm.lab.eng.brq.redhat.com
>>>>>>> Right node: vm-175.idm.lab.eng.brq.redhat.com
>>>>>>> Connectivity: both
>>>>>>> ----------------------------
>>>>>>> Number of entries returned 3
>>>>>>> ----------------------------
>>>>>>> [10:56:34]ofayans at vm-244:~]$
>>>>>>>
>>>>>>> Conclusion:
>>>>>>> We either should completely prohibit the removal of the middle
>>>>>>> nodes (I mean, nodes that hide another active nodes),
>>>>>>> or at the removal stage first recalculate the resulting topology
>>>>>>> and send it to all nodes before actual removal.
>>>>>>> --
>>>>>>> Oleg Fayans
>>>>>>> Quality Engineer
>>>>>>> FreeIPA team
>>>>>>> RedHat.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Oleg Fayans
>>>>> Quality Engineer
>>>>> FreeIPA team
>>>>> RedHat.
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>> --
>>> Oleg Fayans
>>> Quality Engineer
>>> FreeIPA team
>>> RedHat.
>>>
>>>
>>
>>
>>
>
> --
> Oleg Fayans
> Quality Engineer
> FreeIPA team
> RedHat.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/freeipa-devel/attachments/20150624/de126eaf/attachment.htm>
More information about the Freeipa-devel
mailing list