[Freeipa-devel] topology management question

[Simo Sorce]

On Thu, 04 Dec 2014 14:33:09 +0100
Ludwig Krispenz <lkrispen at redhat.com> wrote:

> hi,
> 
> I just have another (hopefully this will end soon) issue I want to
> get your input. (please read to teh end first)
> 
> To recapture the conditions:
> -  the topology plugin manages the connections between servers as 
> segments in the shared tree
> - it is authoritative for managed servers, eg it controls all 
> connections between servers listed under cn=masters,
>    it is permissive for connection to other servers
> - it rejects any removal of a segment, which would disconnect the
> topology.
> - a change in topology can be applied to any server in the topology,
> it will reach the respective servers and the plugin will act upon it
> 
> Now there is a special case, causing a bit of trouble. If a replica
> is to be removed from the topology, this means that
> the replication agreements from and to this replica should be
> removed, the server should be removed from the manages servers.
> The problem is that:
> - if you remove the server first, the server becomes unmanaged and 
> removal of the segment will not trigger a removal of the replication 
> agreement

Can you explain what you mean "if you remove the server first" exactly ?
What LDAP operation will be performed, by the management tools ?

> - if you remove the segments first, one segment will be the last one 
> connecting this replica to the topology and removal will be rejected

We should never remove the segments first indeed.

> Now, with some effort this can be resolved, eg
> if the server is removed, keep it internally as removed server and
> for segments connecting this server trigger removal of replication
> agreements or mark a the last segment, when tried to remove, as
> pending and once the server is removed also remove the corresponding
> repl agreements

Why should we "keep it internally" ?
If you mark the agreements as managed by setting an attribute on them,
then you will never have any issue recognizing a "managed" agreement in
cn=config, and you will also immediately find out it is "old" as it is
not backed by a segment so you will safely remove it.

Segments (and their agreements) should be removed as trigger on the
master entry getting removed. This should be done even if it causes a
split brain, because if the server is removed, no matter how much we
wish to keep tropology integrity we effectively are in a split brain
situation, keeping toplogy agreements alive w/o the server entry
doesn't help.

> But there is a problem, which I think is much harder and I am not
> sure how much effort I should put in resolving it.
> If we want to have the replication agreements cleaned up after
> removal of a replica without direct modification of cn=config, we
> need to follow the path above,
> but this also means that the last change needs to reach both the
> removed replica (R) and the last server(S) it is connected to.

It would be nice if the changed reached the replica, indeed, but not a
big deal if it doesn't, if you are removing the replica it means you
are decommissioning it, so it is not really that important that it
receives updates, it will be destroyed shortly.
And if it is not destroyed for whatever reason, it will be removed from
the masters group anyway so it will have no permission to replicate
back, and no harm is done to the overall domain.

> The bad thing is that if this change triggers a
> removal of the replication agreement on S it could be that the change
> is not replicated to R before the agreement is removed and is lost.
> There is no way (or no easy) way to know for teh plugin if a change
> was received by an other server,

There is an easy way, contact the other server and see if the change
happened in its LDAP tree :)
BNut this is not really necessary, as explained above.

> I was also thinking about some kind
> of acknowledge mechanism by doing a ping pong of changes, but the
> problem always is the same that one server does not know if the other
> has received it.
> And even if this would theoretically work, we cannot be sure that R
> is not shutdown and only the remaining topology is tried to be
> cleaned up, so S would wait forever.

We should not care, if you are deleting a replica it doesn't matter
what's on the replica side IMO.

> My suggestion to resolve this (in most cases) is to define a wait 
> interval, after the final combination of removal of a server and its 
> connecting segment is received, wait for some time and then remove
> the corresponding replication agreements.

Why ?

> So I'm asking you if this would be acceptable or if you have a better 
> solution.

I am trying to understand why we have a problem, actually, I do not
really see one, why do you think it is important to update a replica
that is being killed ?

Simo.

-- 
Simo Sorce * Red Hat, Inc * New York