Routed network can't reach outside network

Thu Jul 23 14:54:18 UTC 2020

On Thu, Jul 23, 2020 at 03:34:03PM +0100, Rui Correia wrote:
> > Is net.ipv4.ip_forward  set to 0 ?
> >
> >
> I assume you're asking if this is setup on the host and not on the VM's.
> I've checked the host and it is configured like this:
> $ sysctl net.ipv4.ip_forward
> net.ipv4.ip_forward = 1

This is good.

> Should I change it to =0 ? It wouldn't make sense to me if I'd change it to
> =0 because that way IP Forwarding would be disabled and not enabled as I
> think enabled is what is needed in this particular case.

No, don't change it to 0.  We need ip_forward enabled as you say.

> Also, for this to work fully, your router switch needs to know where to
> > send traffic for the 10.2.2.1/24 network. IOW, you need to configure
> > static routes on your router switch for 10.2.2.1
> >
> 
> I see. So, let me get this straight. What you're saying is that in order
> for WAN/LAN traffic to reach my VM's I need to set up static IP routes on
> my router - which is the default gateway for the network - to let it know
> how to route the packets to the VM's, which needs to be setup to go through
> the KVM host. Right?
> 
> But, for testing purposes (trying to reach the VM's from the KVM host) I
> don't need those static routes, right?

Correct.  The KVM host knows where the 10.2.2.1/24 subnet is - it owns
it.  The other hosts on your LAN don't know anything about 10.2.2.1/24,
so if they try to access VMs on that subnet, traffic will go to the
default route, aka your LAN router. It then has to know which KVM host
has the 10.2.2.1/24 subnet to send the traffic onwards.

> Because right now I'd be ok if I could reach the VM's from the KVM host and
> right now I can't.

Yep, so that suggests a more fundamental problem with the KVM host
config.

Since ip_forward is confirmed as set to 1,  I feel the most likely problem
is something related to the firewall rules.

Libvirt will create iptables rules to allow traffic. Tradititionally
this would have been sufficient, in iptables all rules are in the single
set of global tables.

If your OS distro has enabled "nft" to replace iptables though, things
become more tricky. In nft world there is no single set of global tables.
Any app using nft can define its own top level tables.

So while libvirt adds iptables rules to allow traffic, there is the
possibility that "nft" rules may none the less deny traffic.

In the specific case of distros using "firewalld", libvirt does some
further workarounds for this problem.

Overall though, I'd be investigating your firewall.

It is helpful to add logging rules to your firewall immediately before
any REJECT / DROP rules so you can spot packets being dropped. That
combined with tcpdump on the TAP devices should help you confirm
what is happening to traffic.

> 
> I've covered a similar setup to yours here:
> >
> >
> > https://www.berrange.com/posts/2009/12/13/routed-subnets-without-nat-for-libvirt-managed-virtual-machines-in-fedora/
> >
> >
> I've looked at your article and I can see that  it resembles my setup. But
> I think I've got that covered.
> This is what the xml file for the virtual network 'routed' looks like:
> 
> <network>
>   <name>routed</name>
>   <uuid>970a25f7-29b6-4a6b-b890-f593eae4fc15</uuid>
>   <forward dev="wlo1" mode="route">
>     <interface dev="wlo1"/>
>   </forward>
>   <bridge name="virbr2" stp="on" delay="0"/>
>   <mac address="52:54:00:bf:35:42"/>
>   <domain name="routed"/>
>   <ip address="10.2.2.1" netmask="255.255.255.0">
>     <dhcp>
>       <range start="10.2.2.11" end="10.11.22.254"/>
>     </dhcp>
>   </ip>
> </network>
> 
> I've got this 'routed' network in use for the virtual nic of my VM's.
> Weird this is, when I run a net-list command, it comes back empty...
> 
> $ virsh net-list
>  Name   State   Autostart   Persistent
> ----------------------------------------
> 
> $
> 
> This doesn't look ok, right? I mean, running a net-list command it should
> list all the networks I have set up on this server. But it's coming back
> empty as if there weren't any networks. I've double checked on Virtual
> Machine Manager and both the 'default' and 'routed' networks are active and
> are set up to 'auto start on boot'.
> Could this somehow be related to my problem? I guess not because the
> 'default' network is the NAT network that I've used before and it worked
> and still works when I set the VM's to use it.

Often missed is that there are multiple instances of libvirtd. One global
instance that runs as root for privileged set ups, and then one per
user instance that runs unprivileged.

If you run "virsh" as non-root, you'll be querying the per-user instance.

virt-manager uses the privileged instance by default.

Try   'virsh -c qemu:///system netlist' instead, or simply run
virsh as root.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|