[libvirt-users] Two Node Cluster

Fri Jun 1 07:54:35 UTC 2018

I have discovered the solution to my problem:
- I still need a NAT type guest network (rather than an "open" or "route"
type).
- I discovered the *iptables rules* that was causing me all this trouble,
and deleted them and everything works as desired.

For a "more permanent solution", I have used a libvirt network hook (
https://www.libvirt.org/hooks.html ) to remove the iptables rule on certain
libvirt network events.
If someone else in the future comes along with a similar problem, see here
for the hook script:
https://gist.github.com/cbluth/61753e8b2568d08294efe556e55246bf#file-etc_libvirt_hooks_network

I have a general question for the mailing list regarding the libvirt
network hook:
As this hook is called on certain events, it deletes the iptables rule.
But, how did the rules get there in the first place? Would there be a race
condition between the event of deleting the iptables rule and the event of
libvirtd creating the iptables rule? Is there a possibility of rejected
packets because of this potential race condition?
I have searched for some info regarding trying to prevent the rules from
being applied in the first place, but I havent found much (or perhaps I am
using the wrong search terms).

Anyways, my original issue is fixed, but any info regarding my questions
above is appreciated.
Thanks,
-Cobin

On Thu, May 31, 2018 at 3:59 PM Cobin Bluth <cbluth at gmail.com> wrote:

> Hi Peter and other Libvirt-Users,
>
> Thank you, I greatly appreciate your response. The diagram I provided may
> be a little misleading, so I have attached a new diagram (also available
> here: https://i.imgur.com/X3nFMCz.png ). I think I also need to provide
> some more detail about my scenario here:
>
> These physical nodes are provided to me by "bare-metal cloud" providers
> like Scaleway / OVH / Packet.net / etc.
> I have opted to use Ubuntu on each bare-metal server, but I am also
> well-versed in RHEL/CentOS environments, so if you want to help me out by
> telling me CentOS-specific details, I'm sure that I can "translate" that to
> an Ubuntu/Debian setup. Considering my scenario with bare-metal providers,
> I do *not* have the flexibility to add/remove physical NICs as I need
> them, I am sort of "locked in" to a single-IP/single-NIC configuration for
> each physical node.
>
> But, I **do** have the flexibility of adding more physical nodes as I
> need them, and I would like to leverage this option when I want to expand
> my cluster. Its easy for me to buy a bare-metal cloud server (takes a few
> minutes), and setting up a 1U/2U server by myself on a "lab" network could
> take me a few hours or so (not to mention, I dont have the resources to
> host my own network for this). Also, I need to account for the fact that
> hardware *will* fail somehow eventually, so adding/removing nodes would
> need to be "easy" in the way that I shouldnt have to configure new DHCP
> scopes on all the other nodes when I decide to add one more node to the
> cluster. Also, as a side note, I do not intend to have any central or
> shared volume pool between the nodes, all storage will be direct on a local
> LVM volume, and the ability to perform live migrations of guests between
> hosts is not a high priority for me, I intend for each guest to be *"stationary"
> to each host*.
>
> Each node's *eth0* is connected via IPSec (unicast and transport mode) to
> the other. This is to encrypt the communication between all nodes because
> it travels over the internet. Additionally, each node currently has a
> *vxlan0* interface in the *172.20.0.0/24 <http://172.20.0.0/24>* network
> which allows them to be all on the same "LAN". I have attached a simple
> diagram illustrating this setup, also available here:
> https://i.imgur.com/jc7bc6b.png . I would *prefer* to use the DHCP that
> comes "built-in" to livbirt, but I am open to running DHCP on each host, or
> as a guest inside each guest network.
>
> So, I hope this sheds a little more light on my scenario.
> If you look at the diagram attached, you can see some *red* lines and
> some question marks. These red lines are the only part of the environment
> that needs to be configured, everything else seems to work fine; KVM can
> provision guests, IPSec tunnel is working, vxlan seems to route/function as
> I think I would need it, guests can reach the internet, and guests can ping
> everything else inside the environment, *except* guests1 cannot ping
> guests2.
>
> Which makes me question the configuration I have for each guest network.
> Each guest network is currently configured with NAT Forwarding, but
> because KVMHost1 cannot ping guests on KVMHost2, I am concluding that a
> NAT'd guest network is most likely something that I do not want. I have
> some choices on how to configure each guest network, it could be an "open
> <https://www.redhat.com/archives/libvir-list/2016-August/msg00640.html>"
> or "routed <https://wiki.libvirt.org/page/VirtualNetworking#Routed_mode>"
> network, but I would still need to configure some static routes.
>
> I will take a stab at doing an open or routed network today, but in the
> meantime, your thoughts and comments about my setup are still welcome and
> encouraged.
>
> Thank you!
> -Cobin
>
>
>
> On Wed, May 30, 2018 at 10:44 PM Peter Crowther <
> peter.crowther at melandra.com> wrote:
>
>> I have to say that I wouldn't do the networking that way - in fact, in
>> the clusters I manage, we haven't done the networking that way :-).  Rather
>> than layer 3 routing between VMs, we've chosen to use layer 2 virtual
>> switching (yes, using openvswitch).  We have the luxury of multiple 10G
>> NICs between our hosts, so we've separated out the management network from
>> the guest network, simply to ensure that we retain administrative access to
>> the hosts via ssh.  If you want to live a little more dangerously, you
>> could use VLAN or VXLAN on one NIC - or you could spend a few dollars on an
>> extra network card on each host for the peace of mind!
>>
>> For the project that's been live for two years: we presently run four
>> hosts on the lab's production network (another two on its acceptance-test
>> network, and another one as a "kickaround" host for playing about with
>> configs).  Guest VMs on all four production hosts share 192.168.59.0/24
>> (why "59" is a story for another day), on an OVS virtual switch on each
>> host named br-guest, with the guest-specific NIC also set as a port on the
>> virtual switch.  Guest traffic is therefore sent transparently between the
>> hosts where needed, and we can live-migrate a guest from one host to
>> another with no need to change the guest's IP address.  Because we share a
>> common guest network and IP range between all hosts, it's trivial to add
>> (or remove) hosts - no host needs to know anything about routing to another
>> host, and in fact only our management layer cares how many hosts we have.
>>
>> We happen to have "controller" nodes that run redundant DHCP servers with
>> non-overlapping scopes, but the exact location is not a requirement of this
>> setup.  We could equally well set up a DHCP service on the guest network on
>> each host, allowing allocation of e.g. 192.168.59.1 to .100 on one host,
>> .101 to .200 on another host.  Guests will typically receive offers from
>> each DHCP server and can choose, which is fine as they're all on the same
>> network.  This provides redundancy in case of a full or failed DHCP server,
>> which your routed network approach wouldn't without some careful DHCP
>> forwarding work.
>>
>> We happen to base our hosts on CentOS 7, but I manage other
>> Debian-derived systems and can probably remember enough about its network
>> setup to help with Ubuntu.  Certainly I can help with OVS weirdnesses; it
>> took some time to get my head round exactly how it works.  That said, I've
>> never set up a kvm host on Debian.
>>
>> Good luck; happy to provide further pointers if useful.
>>
>> Cheers,
>>
>> - Peter
>>
>> On 30 May 2018 at 15:32, Cobin Bluth <cbluth at gmail.com> wrote:
>>
>>> Hello Libvirt Users,
>>>
>>> I would like to setup a two node bare-metal cluster. I need to guidance
>>> on the network configuration. I have attached a small diagram, the same
>>> diagram can be seen here: https://i.imgur.com/SOk6a6G.png
>>>
>>> I would like to configure the following details:
>>> - Each node has a DHCP enabled guest network where VMs will run. (eg, *192.168.1.0/24
>>> <http://192.168.1.0/24>* for Host1, and *192.168.2.0/24
>>> <http://192.168.2.0/24>* for Host2)
>>> - Any guest in Host1 should be able to ping guests in Host2, and vice
>>> versa.
>>> - All guests have routes to reach the open internet (so that '*yum
>>> update*' will work "out-of-the-box")
>>> - Each node will be able to operate fully if the other physical node
>>> fails. (no central DHCP server, etc)
>>> - I would like to *add more physical nodes later* when I need the
>>> resources.
>>>
>>> This is what I have done so far:
>>> - Installed latest Ubuntu 18.04, with latest version of libvirt and
>>> supporting software from ubuntu's apt repo.
>>> - Each node can reach the other via its own eth0.
>>> - Each node has a working vxlan0, which can ping the other via its
>>> vxlan0, so it looks like the vxlan config is working. (I used *ip link
>>> add vxlan0 type vxlan...*)
>>> - Configured route on Host1 like so: *ip route add 192.168.2.0/24
>>> <http://192.168.2.0/24> via 172.20.0.1*
>>> - Configured route on Host2 also: *ip route add 192.168.1.0/24
>>> <http://192.168.1.0/24> via 172.20.0.2*
>>> - All guests on Host1 (and Host1) can ping eth0 and vxlan0 on Host2, and
>>> vice versa, yay.
>>> - Guests on Host1 *cannot* ping guests on Host2, I suspect because the
>>> the default NAT config of the libvirt network.
>>>
>>> So, at this point I started to search for tutorials or more
>>> information/documentation, but I am a little overwhelmed by the sheer
>>> amount of information, as well as a lot of "stale" information on blogs etc.
>>> I have learned that I can *virsh net-edit default*, and then change it
>>> to an "open" network:* <forward mode='open'/>*
>>> After doing this, the guests cannot reach outside their own network, nor
>>> reach the internet, so I assume that I would need to add some routes, or
>>> something else to get the network functioning like I want it. There is also *<forward
>>> mode="route"/>*, but I dont fully understand the scenarios where one
>>> would need an *open* or a *route* forward mode. I have also shied away
>>> from using openvswitch, and have opted for ifupdown2.
>>> (I have taken most of my inspiration from this blog post:
>>> https://joejulian.name/post/how-to-configure-linux-vxlans-with-multiple-unicast-endpoints/
>>>  )
>>>
>>> Some questions that I have for the mailing list, any help would be
>>> greatly appreciated:
>>> - Is my target configuration of a KVM cluster uncommon? Do you see
>>> drawbacks of this setup, or does it go against "typical convention"?
>>> - Would my scenario be better suited for an "*open*" network or a "
>>> *route*" network?
>>> - What would be the approach to complete this setup?
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> libvirt-users mailing list
>>> libvirt-users at redhat.com
>>> https://www.redhat.com/mailman/listinfo/libvirt-users
>>>
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/libvirt-users/attachments/20180601/67d83073/attachment.htm>