<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<br>
<div class="moz-cite-prefix">On 06/24/2015 12:50 PM, Oleg Fayans
wrote:<br>
</div>
<blockquote cite="mid:558A8B87.5020203@redhat.com" type="cite">
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
<br>
<br>
<div class="moz-cite-prefix">On 06/24/2015 12:28 PM, Ludwig
Krispenz wrote:<br>
</div>
<blockquote cite="mid:558A8660.6070905@redhat.com" type="cite">
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
<br>
<div class="moz-cite-prefix">On 06/24/2015 12:02 PM, Oleg Fayans
wrote:<br>
</div>
<blockquote cite="mid:558A801E.5010906@redhat.com" type="cite">
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
<br>
<br>
<div class="moz-cite-prefix">On 06/24/2015 11:47 AM, Ludwig
Krispenz wrote:<br>
</div>
<blockquote cite="mid:558A7CA5.1090301@redhat.com" type="cite">
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
<br>
<div class="moz-cite-prefix">On 06/24/2015 11:36 AM, Oleg
Fayans wrote:<br>
</div>
<blockquote cite="mid:558A7A09.9060408@redhat.com"
type="cite">
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
<br>
<br>
<div class="moz-cite-prefix">On 06/24/2015 11:25 AM,
Ludwig Krispenz wrote:<br>
</div>
<blockquote cite="mid:558A7798.8020000@redhat.com"
type="cite">
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
Oleg,<br>
<br>
the topology plugin relies on existing connection
between servers which remain in a topolgy. If you remove
a central node in your topology you are asking for
trouble.<br>
With Petr's patch it warns you that your topology will
be disconnected, and if you insist we cannot guarantee
anything.<br>
</blockquote>
Agree. I just wanted to try edge cases to see how one can
break the system :)
<blockquote cite="mid:558A7798.8020000@redhat.com"
type="cite"> should we completely prohibit this ? I
don't know, I think you could also enforce an uninstall
of vm175 with probably the same result.<br>
what you mean be calculating the remaining topology and
send it to the remaining servers does not work, it would
require to send a removal of a segment, which would be
rejected.<br>
<br>
The topology is broken, and I don't know how much we
should invest in making this info consistent on all
servers. <br>
<br>
More interesting would be if we can heal this later by
adding new segments.<br>
</blockquote>
Yes, here comes the biggest question raised from this
case: obviously, when none of the nodes possess the
correct topology information (including the one which
deleted the central node), there is no way to fix it by
adding segments connecting the nodes that became
disconnected. </blockquote>
It shoul not need the full information, but it has to be
able to reach one of the nodes to be connected. when the
topology is broken, you loose to feature to be ably to apply
a change on any node, eg in your case if you want to connect
vm036 and vm056 an have removed vm175, you have to do it on
vm056, vm036 or vm244. This should work, if not we have to
fix it - unless we completely prevent disconnecting a
topology<br>
</blockquote>
Well, this is exactly the problem here: all replicas should
contain precise copies of all the info: accounts, hosts,
sudorules, etc, including topology information. However, if in
this case I manually connect disconnected node at vm127 (or
vm056, does not matter) it results in topology information
inconsistency across the infrastructure:<br>
This would be the topology from the point of view of vm127:<br>
</blockquote>
did you add teh connection on vm127 or on vm244 ? sorry, but in
these situations to understand what's going on, it can matter. <br>
to me it looks like you did it on vm127, so its there, it got
replicated to vm244, but replicationback does not work and so
the deletion of teh segs to vm175, which should still be in the
changelogs of 036 and 244, don#t get to 127. Do you have
something in the error logs of 244 ?<br>
</blockquote>
Yes, I added the connection on vm127. vm244 does not have anything
in the ldap errors log corresponding to the replication with
vm127. In fact, I tried to create a user on vm244 to see if it
will be replicated to vm127, and the user creation failed with the
following error message:<big><big><big><span style="color: rgb(51,
51, 51); font-family: 'Open Sans', Helvetica, Arial,
sans-serif; font-size: 12px; font-style: normal;
font-variant: normal; font-weight: normal; letter-spacing:
normal; line-height: 20px; orphans: auto; text-align:
start; text-indent: 0px; text-transform: none;
white-space: normal; widows: 1; word-spacing: 0px;
-webkit-text-stroke-width: 0px; display: inline
!important; float: none; background-color: rgb(255, 255,
255);"></span></big></big></big><br>
Operations error: Allocation of a new value for range cn=posix
ids,cn=distributed numeric assignment plugin,cn=plugins,cn=config
failed! Unable to proceed.<br>
<br>
Is it because the master node was deleted?<br>
</blockquote>
think so, yes. <br>
There are probably more things to check before removing a server :-(<br>
<br>
<blockquote cite="mid:558A8B87.5020203@redhat.com" type="cite"> The
corresponding message in the error log is <br>
[24/Jun/2015:12:44:18 +0200] dna-plugin - dna_pre_op: no more
values available!!<br>
<blockquote cite="mid:558A8660.6070905@redhat.com" type="cite"> <br>
<blockquote cite="mid:558A801E.5010906@redhat.com" type="cite">
<br>
vm056 vm036<br>
\ / |<br>
vm175 |<br>
\ |<br>
vm127 vm244<br>
<br>
And this - from the point of view of vm244 and vm036<br>
<br>
vm056 vm036<br>
\ |<br>
vm175 |<br>
|<br>
vm127 ----- vm244<br>
<blockquote cite="mid:558A7CA5.1090301@redhat.com" type="cite">
<blockquote cite="mid:558A7A09.9060408@redhat.com"
type="cite">I still think that the recalculation of the
resulting tree should be done at least on the node that
performs the removal action. And when later some other
node gets connected, it should understand somehow that
it's topology information is outdated<br>
<blockquote cite="mid:558A7798.8020000@redhat.com"
type="cite"> <br>
Ludwig<br>
<div class="moz-cite-prefix">On 06/24/2015 11:04 AM,
Oleg Fayans wrote:<br>
</div>
<blockquote cite="mid:558A729F.1030307@redhat.com"
type="cite">
<meta http-equiv="content-type" content="text/html;
charset=ISO-8859-1">
Hi everybody,<br>
<br>
Current implementation of topology plugin (including
patch 878 from Petr) allows the deletion of the
central node in the star topology.<br>
I had the following topology:<br>
<br>
vm056 vm036<br>
\ / |<br>
vm175 |<br>
/ \ |<br>
vm127 vm244<br>
<br>
I was able to remove node vm175 from node vm244:<br>
<br>
[17:54:48]ofayans@vm-244:~]$ ipa-replica-manage del
vm-175.idm.lab.eng.brq.redhat.com <br>
Topology after removal of
vm-175.idm.lab.eng.brq.redhat.com will be
disconnected:<br>
Server vm-036.idm.lab.eng.brq.redhat.com can't contact
servers: vm-056.idm.lab.eng.brq.redhat.com,
vm-127.idm.lab.eng.brq.redhat.com<br>
Server vm-056.idm.lab.eng.brq.redhat.com can't contact
servers: vm-244.idm.lab.eng.brq.redhat.com,
vm-036.idm.lab.eng.brq.redhat.com,
vm-127.idm.lab.eng.brq.redhat.com<br>
Server vm-127.idm.lab.eng.brq.redhat.com can't contact
servers: vm-244.idm.lab.eng.brq.redhat.com,
vm-056.idm.lab.eng.brq.redhat.com,
vm-036.idm.lab.eng.brq.redhat.com<br>
Server vm-244.idm.lab.eng.brq.redhat.com can't contact
servers: vm-056.idm.lab.eng.brq.redhat.com,
vm-127.idm.lab.eng.brq.redhat.com<br>
Continue to delete? [no]: yes<br>
Waiting for removal of replication agreements<br>
unexpected error: limits exceeded for this query<br>
<br>
I would expect this operation to delete 4 replication
agreements on all nodes:<br>
vm056 - vm175<br>
vm127 - vm175<br>
vm244 - vm175<br>
vm036 - vm175<br>
<br>
However an arbitrary set of replication agreements was
deleted on each node leading to total infrastructure
inconsistency:<br>
===============================================================<br>
vm056<b> </b>thought the topology was as follows:<br>
vm056 vm036<br>
/ |<br>
vm175 |<br>
/ \ |<br>
vm127 vm244<br>
[10:28:55]ofayans@vm-056:~]$ ipa topologysegment-find
realm<br>
------------------<br>
4 segments matched<br>
------------------<br>
Segment name: 036-to-244<br>
Left node: vm-036.idm.lab.eng.brq.redhat.com<br>
Right node: vm-244.idm.lab.eng.brq.redhat.com<br>
Connectivity: both<br>
<br>
Segment name:
vm-036.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com<br>
Left node: vm-036.idm.lab.eng.brq.redhat.com<br>
Right node: vm-175.idm.lab.eng.brq.redhat.com<br>
Connectivity: both<br>
<br>
Segment name:
vm-127.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com<br>
Left node: vm-127.idm.lab.eng.brq.redhat.com<br>
Right node: vm-175.idm.lab.eng.brq.redhat.com<br>
Connectivity: both<br>
<br>
Segment name:
vm-175.idm.lab.eng.brq.redhat.com-to-vm-244.idm.lab.eng.brq.redhat.com<br>
Left node: vm-175.idm.lab.eng.brq.redhat.com<br>
Right node: vm-244.idm.lab.eng.brq.redhat.com<br>
Connectivity: both<br>
----------------------------<br>
Number of entries returned 4<br>
----------------------------<br>
===============================================================<br>
both vm036<b> </b>vm244 thought the topology was as
follows:<br>
vm056 vm036<br>
\ |<br>
vm175 |<br>
/ |<br>
vm127 vm244<br>
<br>
[10:26:23]ofayans@vm-036:~]$ ipa topologysegment-find<br>
Suffix name: realm<br>
------------------<br>
3 segments matched<br>
------------------<br>
Segment name: 036-to-244<br>
Left node: vm-036.idm.lab.eng.brq.redhat.com<br>
Right node: vm-244.idm.lab.eng.brq.redhat.com<br>
Connectivity: both<br>
<br>
Segment name:
vm-056.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com<br>
Left node: vm-056.idm.lab.eng.brq.redhat.com<br>
Right node: vm-175.idm.lab.eng.brq.redhat.com<br>
Connectivity: both<br>
<br>
Segment name:
vm-127.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com<br>
Left node: vm-127.idm.lab.eng.brq.redhat.com<br>
Right node: vm-175.idm.lab.eng.brq.redhat.com<br>
Connectivity: both<br>
----------------------------<br>
Number of entries returned 3<br>
----------------------------<br>
<br>
===============================================================<br>
<b> </b>vm127 thought the topology was as follows:<br>
vm056 vm036<br>
\ / |<br>
vm175 |<br>
\ |<br>
vm127 vm244<br>
<br>
[10:31:08]ofayans@vm-127:~]$ ipa topologysegment-find
realm<br>
------------------<br>
4 segments matched<br>
------------------<br>
Segment name: 036-to-244<br>
Left node: vm-036.idm.lab.eng.brq.redhat.com<br>
Right node: vm-244.idm.lab.eng.brq.redhat.com<br>
Connectivity: both<br>
<br>
Segment name:
vm-036.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com<br>
Left node: vm-036.idm.lab.eng.brq.redhat.com<br>
Right node: vm-175.idm.lab.eng.brq.redhat.com<br>
Connectivity: both<br>
<br>
Segment name:
vm-056.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com<br>
Left node: vm-056.idm.lab.eng.brq.redhat.com<br>
Right node: vm-175.idm.lab.eng.brq.redhat.com<br>
Connectivity: both<br>
<br>
Segment name:
vm-175.idm.lab.eng.brq.redhat.com-to-vm-244.idm.lab.eng.brq.redhat.com<br>
Left node: vm-175.idm.lab.eng.brq.redhat.com<br>
Right node: vm-244.idm.lab.eng.brq.redhat.com<br>
Connectivity: both<br>
----------------------------<br>
Number of entries returned 4<br>
----------------------------<br>
<br>
If I, for example, add a segment connecting vm127 and
vm244, these two nodes will not synchronize the
topology info:<br>
<br>
[10:51:03]ofayans@vm-127:~]$ ipa topologysegment-add
realm 127-to-244
--leftnode=vm-127.idm.lab.eng.brq.redhat.com
--rightnode=vm-244.idm.lab.eng.brq.redhat.com
--direction=both<br>
--------------------------<br>
Added segment "127-to-244"<br>
--------------------------<br>
Segment name: 127-to-244<br>
Left node: vm-127.idm.lab.eng.brq.redhat.com<br>
Right node: vm-244.idm.lab.eng.brq.redhat.com<br>
Connectivity: both<br>
[10:53:33]ofayans@vm-127:~]$ ipa topologysegment-find
realm<br>
------------------<br>
5 segments matched<br>
------------------<br>
Segment name: 036-to-244<br>
Left node: vm-036.idm.lab.eng.brq.redhat.com<br>
Right node: vm-244.idm.lab.eng.brq.redhat.com<br>
Connectivity: both<br>
<br>
Segment name: 127-to-244<br>
Left node: vm-127.idm.lab.eng.brq.redhat.com<br>
Right node: vm-244.idm.lab.eng.brq.redhat.com<br>
Connectivity: both<br>
<br>
Segment name:
vm-036.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com<br>
Left node: vm-036.idm.lab.eng.brq.redhat.com<br>
Right node: vm-175.idm.lab.eng.brq.redhat.com<br>
Connectivity: both<br>
<br>
Segment name:
vm-056.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com<br>
Left node: vm-056.idm.lab.eng.brq.redhat.com<br>
Right node: vm-175.idm.lab.eng.brq.redhat.com<br>
Connectivity: both<br>
<br>
Segment name:
vm-175.idm.lab.eng.brq.redhat.com-to-vm-244.idm.lab.eng.brq.redhat.com<br>
Left node: vm-175.idm.lab.eng.brq.redhat.com<br>
Right node: vm-244.idm.lab.eng.brq.redhat.com<br>
Connectivity: both<br>
----------------------------<br>
Number of entries returned 5<br>
----------------------------<br>
[10:54:02]ofayans@vm-127:~]$ <br>
<br>
=============================================================<br>
<br>
[10:49:38]ofayans@vm-244:~]$ ipa topologysegment-find
realm<br>
------------------<br>
3 segments matched<br>
------------------<br>
Segment name: 036-to-244<br>
Left node: vm-036.idm.lab.eng.brq.redhat.com<br>
Right node: vm-244.idm.lab.eng.brq.redhat.com<br>
Connectivity: both<br>
<br>
Segment name: 127-to-244<br>
Left node: vm-127.idm.lab.eng.brq.redhat.com<br>
Right node: vm-244.idm.lab.eng.brq.redhat.com<br>
Connectivity: both<br>
<br>
Segment name:
vm-056.idm.lab.eng.brq.redhat.com-to-vm-175.idm.lab.eng.brq.redhat.com<br>
Left node: vm-056.idm.lab.eng.brq.redhat.com<br>
Right node: vm-175.idm.lab.eng.brq.redhat.com<br>
Connectivity: both<br>
----------------------------<br>
Number of entries returned 3<br>
----------------------------<br>
[10:56:34]ofayans@vm-244:~]$ <br>
<br>
<big>Conclusion:</big><br>
We either should completely prohibit the removal of
the middle nodes (I mean, nodes that hide another
active nodes),<br>
or at the removal stage first recalculate the
resulting topology and send it to all nodes before
actual removal.<br>
<pre class="moz-signature" cols="72">--
Oleg Fayans
Quality Engineer
FreeIPA team
RedHat.</pre>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
</blockquote>
<br>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Oleg Fayans
Quality Engineer
FreeIPA team
RedHat.</pre>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
</blockquote>
<br>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Oleg Fayans
Quality Engineer
FreeIPA team
RedHat.</pre>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
</blockquote>
<br>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Oleg Fayans
Quality Engineer
FreeIPA team
RedHat.</pre>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
</blockquote>
<br>
</body>
</html>