[libvirt-users] Two Node Cluster

Thu May 31 13:59:32 UTC 2018

Hi Peter and other Libvirt-Users,

Thank you, I greatly appreciate your response. The diagram I provided may
be a little misleading, so I have attached a new diagram (also available
here: https://i.imgur.com/X3nFMCz.png ). I think I also need to provide
some more detail about my scenario here:

These physical nodes are provided to me by "bare-metal cloud" providers
like Scaleway / OVH / Packet.net / etc.
I have opted to use Ubuntu on each bare-metal server, but I am also
well-versed in RHEL/CentOS environments, so if you want to help me out by
telling me CentOS-specific details, I'm sure that I can "translate" that to
an Ubuntu/Debian setup. Considering my scenario with bare-metal providers,
I do *not* have the flexibility to add/remove physical NICs as I need them,
I am sort of "locked in" to a single-IP/single-NIC configuration for each
physical node.

But, I **do** have the flexibility of adding more physical nodes as I need
them, and I would like to leverage this option when I want to expand my
cluster. Its easy for me to buy a bare-metal cloud server (takes a few
minutes), and setting up a 1U/2U server by myself on a "lab" network could
take me a few hours or so (not to mention, I dont have the resources to
host my own network for this). Also, I need to account for the fact that
hardware *will* fail somehow eventually, so adding/removing nodes would
need to be "easy" in the way that I shouldnt have to configure new DHCP
scopes on all the other nodes when I decide to add one more node to the
cluster. Also, as a side note, I do not intend to have any central or
shared volume pool between the nodes, all storage will be direct on a local
LVM volume, and the ability to perform live migrations of guests between
hosts is not a high priority for me, I intend for each guest to be
*"stationary"
to each host*.

Each node's *eth0* is connected via IPSec (unicast and transport mode) to
the other. This is to encrypt the communication between all nodes because
it travels over the internet. Additionally, each node currently has a
*vxlan0* interface in the *172.20.0.0/24 <http://172.20.0.0/24>* network
which allows them to be all on the same "LAN". I have attached a simple
diagram illustrating this setup, also available here:
https://i.imgur.com/jc7bc6b.png . I would *prefer* to use the DHCP that
comes "built-in" to livbirt, but I am open to running DHCP on each host, or
as a guest inside each guest network.

So, I hope this sheds a little more light on my scenario.
If you look at the diagram attached, you can see some *red* lines and some
question marks. These red lines are the only part of the environment that
needs to be configured, everything else seems to work fine; KVM can
provision guests, IPSec tunnel is working, vxlan seems to route/function as
I think I would need it, guests can reach the internet, and guests can ping
everything else inside the environment, *except* guests1 cannot ping
guests2.

Which makes me question the configuration I have for each guest network.
Each guest network is currently configured with NAT Forwarding, but because
KVMHost1 cannot ping guests on KVMHost2, I am concluding that a NAT'd guest
network is most likely something that I do not want. I have some choices on
how to configure each guest network, it could be an "open
<https://www.redhat.com/archives/libvir-list/2016-August/msg00640.html>" or
"routed <https://wiki.libvirt.org/page/VirtualNetworking#Routed_mode>"
network, but I would still need to configure some static routes.

I will take a stab at doing an open or routed network today, but in the
meantime, your thoughts and comments about my setup are still welcome and
encouraged.

Thank you!
-Cobin

On Wed, May 30, 2018 at 10:44 PM Peter Crowther <peter.crowther at melandra.com>
wrote:

> I have to say that I wouldn't do the networking that way - in fact, in the
> clusters I manage, we haven't done the networking that way :-).  Rather
> than layer 3 routing between VMs, we've chosen to use layer 2 virtual
> switching (yes, using openvswitch).  We have the luxury of multiple 10G
> NICs between our hosts, so we've separated out the management network from
> the guest network, simply to ensure that we retain administrative access to
> the hosts via ssh.  If you want to live a little more dangerously, you
> could use VLAN or VXLAN on one NIC - or you could spend a few dollars on an
> extra network card on each host for the peace of mind!
>
> For the project that's been live for two years: we presently run four
> hosts on the lab's production network (another two on its acceptance-test
> network, and another one as a "kickaround" host for playing about with
> configs).  Guest VMs on all four production hosts share 192.168.59.0/24
> (why "59" is a story for another day), on an OVS virtual switch on each
> host named br-guest, with the guest-specific NIC also set as a port on the
> virtual switch.  Guest traffic is therefore sent transparently between the
> hosts where needed, and we can live-migrate a guest from one host to
> another with no need to change the guest's IP address.  Because we share a
> common guest network and IP range between all hosts, it's trivial to add
> (or remove) hosts - no host needs to know anything about routing to another
> host, and in fact only our management layer cares how many hosts we have.
>
> We happen to have "controller" nodes that run redundant DHCP servers with
> non-overlapping scopes, but the exact location is not a requirement of this
> setup.  We could equally well set up a DHCP service on the guest network on
> each host, allowing allocation of e.g. 192.168.59.1 to .100 on one host,
> .101 to .200 on another host.  Guests will typically receive offers from
> each DHCP server and can choose, which is fine as they're all on the same
> network.  This provides redundancy in case of a full or failed DHCP server,
> which your routed network approach wouldn't without some careful DHCP
> forwarding work.
>
> We happen to base our hosts on CentOS 7, but I manage other Debian-derived
> systems and can probably remember enough about its network setup to help
> with Ubuntu.  Certainly I can help with OVS weirdnesses; it took some time
> to get my head round exactly how it works.  That said, I've never set up a
> kvm host on Debian.
>
> Good luck; happy to provide further pointers if useful.
>
> Cheers,
>
> - Peter
>
> On 30 May 2018 at 15:32, Cobin Bluth <cbluth at gmail.com> wrote:
>
>> Hello Libvirt Users,
>>
>> I would like to setup a two node bare-metal cluster. I need to guidance
>> on the network configuration. I have attached a small diagram, the same
>> diagram can be seen here: https://i.imgur.com/SOk6a6G.png
>>
>> I would like to configure the following details:
>> - Each node has a DHCP enabled guest network where VMs will run. (eg, *192.168.1.0/24
>> <http://192.168.1.0/24>* for Host1, and *192.168.2.0/24
>> <http://192.168.2.0/24>* for Host2)
>> - Any guest in Host1 should be able to ping guests in Host2, and vice
>> versa.
>> - All guests have routes to reach the open internet (so that '*yum
>> update*' will work "out-of-the-box")
>> - Each node will be able to operate fully if the other physical node
>> fails. (no central DHCP server, etc)
>> - I would like to *add more physical nodes later* when I need the
>> resources.
>>
>> This is what I have done so far:
>> - Installed latest Ubuntu 18.04, with latest version of libvirt and
>> supporting software from ubuntu's apt repo.
>> - Each node can reach the other via its own eth0.
>> - Each node has a working vxlan0, which can ping the other via its
>> vxlan0, so it looks like the vxlan config is working. (I used *ip link
>> add vxlan0 type vxlan...*)
>> - Configured route on Host1 like so: *ip route add 192.168.2.0/24
>> <http://192.168.2.0/24> via 172.20.0.1*
>> - Configured route on Host2 also: *ip route add 192.168.1.0/24
>> <http://192.168.1.0/24> via 172.20.0.2*
>> - All guests on Host1 (and Host1) can ping eth0 and vxlan0 on Host2, and
>> vice versa, yay.
>> - Guests on Host1 *cannot* ping guests on Host2, I suspect because the
>> the default NAT config of the libvirt network.
>>
>> So, at this point I started to search for tutorials or more
>> information/documentation, but I am a little overwhelmed by the sheer
>> amount of information, as well as a lot of "stale" information on blogs etc.
>> I have learned that I can *virsh net-edit default*, and then change it
>> to an "open" network:* <forward mode='open'/>*
>> After doing this, the guests cannot reach outside their own network, nor
>> reach the internet, so I assume that I would need to add some routes, or
>> something else to get the network functioning like I want it. There is also *<forward
>> mode="route"/>*, but I dont fully understand the scenarios where one
>> would need an *open* or a *route* forward mode. I have also shied away
>> from using openvswitch, and have opted for ifupdown2.
>> (I have taken most of my inspiration from this blog post:
>> https://joejulian.name/post/how-to-configure-linux-vxlans-with-multiple-unicast-endpoints/
>>  )
>>
>> Some questions that I have for the mailing list, any help would be
>> greatly appreciated:
>> - Is my target configuration of a KVM cluster uncommon? Do you see
>> drawbacks of this setup, or does it go against "typical convention"?
>> - Would my scenario be better suited for an "*open*" network or a "
>> *route*" network?
>> - What would be the approach to complete this setup?
>>
>>
>>
>>
>> _______________________________________________
>> libvirt-users mailing list
>> libvirt-users at redhat.com
>> https://www.redhat.com/mailman/listinfo/libvirt-users
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/libvirt-users/attachments/20180531/a5cf5195/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: X3nFMCz.png
Type: image/png
Size: 79915 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/libvirt-users/attachments/20180531/a5cf5195/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: jc7bc6b.png
Type: image/png
Size: 42413 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/libvirt-users/attachments/20180531/a5cf5195/attachment-0001.png>