[Freeipa-devel] Topology: Central node removal in star topology
Simo Sorce
simo at redhat.com
Wed Jun 24 19:01:26 UTC 2015
On Wed, 2015-06-24 at 11:25 +0200, Ludwig Krispenz wrote:
> Oleg,
>
> the topology plugin relies on existing connection between servers which
> remain in a topolgy. If you remove a central node in your topology you
> are asking for trouble.
> With Petr's patch it warns you that your topology will be disconnected,
> and if you insist we cannot guarantee anything.
> should we completely prohibit this ?
No, but a --force should be needed.
Without a --force option we should not allow to remove a replica
completely from another one.
> I don't know, I think you could
> also enforce an uninstall of vm175 with probably the same result.
> what you mean be calculating the remaining topology and send it to the
> remaining servers does not work, it would require to send a removal of a
> segment, which would be rejected.
You would have to connect to each replica that has a replication
agreement with vm175 and remove the segment from that replica. But it
wouldn't really help much as once a replica is isolated from the central
one, it will not see the other operations going on in other replicas.
Once we have a topology resolver we will be able to warn that removing a
specific replica will cause a split brain and make very loud warnings
and even offer solutions on how to reconnect the remaining replicas, but
nothing else can really be done if the admin insist in break the
replication topology, I guess.
> The topology is broken, and I don't know how much we should invest in
> making this info consistent on all servers.
We just need to make it very clear to the admin that replication is
broken, later on we'll have visual tools to make it easier to understand
what is going on, but that's all we can do.
> More interesting would be if we can heal this later by adding new segments.
Indeed, reconnecting all the severed replicas should cause all the
removals (segments or servers) to be replicated among servers and should
bring back the topology view in a consistent state. But not until all
servers are reconnected and replication has started again.
Simo.
> Ludwig
> On 06/24/2015 11:04 AM, Oleg Fayans wrote:
> > Hi everybody,
> >
> > Current implementation of topology plugin (including patch 878 from
> > Petr) allows the deletion of the central node in the star topology.
> > I had the following topology:
> >
> > vm056 vm036
> > \ / |
> > vm175 |
> > / \ |
> > vm127 vm244
> >
> > I was able to remove node vm175 from node vm244:
> >
> > [17:54:48]ofayans at vm-244:~]$ ipa-replica-manage del
> > vm-175.idm.lab.eng.brq.redhat.com
> > Topology after removal of vm-175.idm.lab.eng.brq.redhat.com will be
> > disconnected:
> > Server vm-036.idm.lab.eng.brq.redhat.com can't contact servers:
> > vm-056.idm.lab.eng.brq.redhat.com, vm-127.idm.lab.eng.brq.redhat.com
> > Server vm-056.idm.lab.eng.brq.redhat.com can't contact servers:
> > vm-244.idm.lab.eng.brq.redhat.com, vm-036.idm.lab.eng.brq.redhat.com,
> > vm-127.idm.lab.eng.brq.redhat.com
> > Server vm-127.idm.lab.eng.brq.redhat.com can't contact servers:
> > vm-244.idm.lab.eng.brq.redhat.com, vm-056.idm.lab.eng.brq.redhat.com,
> > vm-036.idm.lab.eng.brq.redhat.com
> > Server vm-244.idm.lab.eng.brq.redhat.com can't contact servers:
> > vm-056.idm.lab.eng.brq.redhat.com, vm-127.idm.lab.eng.brq.redhat.com
> > Continue to delete? [no]: yes
> > Waiting for removal of replication agreements
> > unexpected error: limits exceeded for this query
> >
> > I would expect this operation to delete 4 replication agreements on
> > all nodes:
> > vm056 - vm175
> > vm127 - vm175
> > vm244 - vm175
> > vm036 - vm175
> >
> > However an arbitrary set of replication agreements was deleted on each
> > node leading to total infrastructure inconsistency:
> > ===============================================================
> > vm056**thought the topology was as follows:
> > vm056 vm036
> > / |
> > vm175 |
> > / \ |
> > vm127 vm244
> > [10:28:55]ofayans at vm-056:~]$ ipa topologysegment-find realm
> > ------------------
> > 4 segments matched
> > ------------------
> > Segment name: 036-to-244
> > Left node: vm-036.idm.lab.eng.brq.redhat.com
> > Right node: vm-244.idm.lab.eng.brq.redhat.com
> > Connectivity: both
> >
> > Segment name:
> > vm-036.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
> > Left node: vm-036.idm.lab.eng.brq.redhat.com
> > Right node: vm-175.idm.lab.eng.brq.redhat.com
> > Connectivity: both
> >
> > Segment name:
> > vm-127.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
> > Left node: vm-127.idm.lab.eng.brq.redhat.com
> > Right node: vm-175.idm.lab.eng.brq.redhat.com
> > Connectivity: both
> >
> > Segment name:
> > vm-175.idm.lab.eng.brq.redhat.com-to-vm-244.idm.lab.eng.brq.redhat.com
> > Left node: vm-175.idm.lab.eng.brq.redhat.com
> > Right node: vm-244.idm.lab.eng.brq.redhat.com
> > Connectivity: both
> > ----------------------------
> > Number of entries returned 4
> > ----------------------------
> > ===============================================================
> > both vm036**vm244 thought the topology was as follows:
> > vm056 vm036
> > \ |
> > vm175 |
> > / |
> > vm127 vm244
> >
> > [10:26:23]ofayans at vm-036:~]$ ipa topologysegment-find
> > Suffix name: realm
> > ------------------
> > 3 segments matched
> > ------------------
> > Segment name: 036-to-244
> > Left node: vm-036.idm.lab.eng.brq.redhat.com
> > Right node: vm-244.idm.lab.eng.brq.redhat.com
> > Connectivity: both
> >
> > Segment name:
> > vm-056.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
> > Left node: vm-056.idm.lab.eng.brq.redhat.com
> > Right node: vm-175.idm.lab.eng.brq.redhat.com
> > Connectivity: both
> >
> > Segment name:
> > vm-127.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
> > Left node: vm-127.idm.lab.eng.brq.redhat.com
> > Right node: vm-175.idm.lab.eng.brq.redhat.com
> > Connectivity: both
> > ----------------------------
> > Number of entries returned 3
> > ----------------------------
> >
> > ===============================================================
> > **vm127 thought the topology was as follows:
> > vm056 vm036
> > \ / |
> > vm175 |
> > \ |
> > vm127 vm244
> >
> > [10:31:08]ofayans at vm-127:~]$ ipa topologysegment-find realm
> > ------------------
> > 4 segments matched
> > ------------------
> > Segment name: 036-to-244
> > Left node: vm-036.idm.lab.eng.brq.redhat.com
> > Right node: vm-244.idm.lab.eng.brq.redhat.com
> > Connectivity: both
> >
> > Segment name:
> > vm-036.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
> > Left node: vm-036.idm.lab.eng.brq.redhat.com
> > Right node: vm-175.idm.lab.eng.brq.redhat.com
> > Connectivity: both
> >
> > Segment name:
> > vm-056.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
> > Left node: vm-056.idm.lab.eng.brq.redhat.com
> > Right node: vm-175.idm.lab.eng.brq.redhat.com
> > Connectivity: both
> >
> > Segment name:
> > vm-175.idm.lab.eng.brq.redhat.com-to-vm-244.idm.lab.eng.brq.redhat.com
> > Left node: vm-175.idm.lab.eng.brq.redhat.com
> > Right node: vm-244.idm.lab.eng.brq.redhat.com
> > Connectivity: both
> > ----------------------------
> > Number of entries returned 4
> > ----------------------------
> >
> > If I, for example, add a segment connecting vm127 and vm244, these two
> > nodes will not synchronize the topology info:
> >
> > [10:51:03]ofayans at vm-127:~]$ ipa topologysegment-add realm 127-to-244
> > --leftnode=vm-127.idm.lab.eng.brq.redhat.com
> > --rightnode=vm-244.idm.lab.eng.brq.redhat.com --direction=both
> > --------------------------
> > Added segment "127-to-244"
> > --------------------------
> > Segment name: 127-to-244
> > Left node: vm-127.idm.lab.eng.brq.redhat.com
> > Right node: vm-244.idm.lab.eng.brq.redhat.com
> > Connectivity: both
> > [10:53:33]ofayans at vm-127:~]$ ipa topologysegment-find realm
> > ------------------
> > 5 segments matched
> > ------------------
> > Segment name: 036-to-244
> > Left node: vm-036.idm.lab.eng.brq.redhat.com
> > Right node: vm-244.idm.lab.eng.brq.redhat.com
> > Connectivity: both
> >
> > Segment name: 127-to-244
> > Left node: vm-127.idm.lab.eng.brq.redhat.com
> > Right node: vm-244.idm.lab.eng.brq.redhat.com
> > Connectivity: both
> >
> > Segment name:
> > vm-036.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
> > Left node: vm-036.idm.lab.eng.brq.redhat.com
> > Right node: vm-175.idm.lab.eng.brq.redhat.com
> > Connectivity: both
> >
> > Segment name:
> > vm-056.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
> > Left node: vm-056.idm.lab.eng.brq.redhat.com
> > Right node: vm-175.idm.lab.eng.brq.redhat.com
> > Connectivity: both
> >
> > Segment name:
> > vm-175.idm.lab.eng.brq.redhat.com-to-vm-244.idm.lab.eng.brq.redhat.com
> > Left node: vm-175.idm.lab.eng.brq.redhat.com
> > Right node: vm-244.idm.lab.eng.brq.redhat.com
> > Connectivity: both
> > ----------------------------
> > Number of entries returned 5
> > ----------------------------
> > [10:54:02]ofayans at vm-127:~]$
> >
> > =============================================================
> >
> > [10:49:38]ofayans at vm-244:~]$ ipa topologysegment-find realm
> > ------------------
> > 3 segments matched
> > ------------------
> > Segment name: 036-to-244
> > Left node: vm-036.idm.lab.eng.brq.redhat.com
> > Right node: vm-244.idm.lab.eng.brq.redhat.com
> > Connectivity: both
> >
> > Segment name: 127-to-244
> > Left node: vm-127.idm.lab.eng.brq.redhat.com
> > Right node: vm-244.idm.lab.eng.brq.redhat.com
> > Connectivity: both
> >
> > Segment name:
> > vm-056.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com
> > Left node: vm-056.idm.lab.eng.brq.redhat.com
> > Right node: vm-175.idm.lab.eng.brq.redhat.com
> > Connectivity: both
> > ----------------------------
> > Number of entries returned 3
> > ----------------------------
> > [10:56:34]ofayans at vm-244:~]$
> >
> > Conclusion:
> > We either should completely prohibit the removal of the middle nodes
> > (I mean, nodes that hide another active nodes),
> > or at the removal stage first recalculate the resulting topology and
> > send it to all nodes before actual removal.
> > --
> > Oleg Fayans
> > Quality Engineer
> > FreeIPA team
> > RedHat.
> >
> >
>
--
Simo Sorce * Red Hat, Inc * New York
More information about the Freeipa-devel
mailing list