<div dir="ltr">Hello,<div><br></div><div>I came across an interesting problem in my home lab a few weeks ago as I'm prepping for my RHCE exam using Michael Jang study guide. I've been at this for days now, and I still can't wrap my head around how two or more virtual networks in default NAT configuration are even allowed to communicate with each other despite what the libvirt documentation said.<br></div><div><br></div><div><br></div><div>Here's the excerpt I'm referring to in the wiki link here: <a href="http://wiki.libvirt.org/page/Networking#Forwarding_Incoming_Connections">http://wiki.libvirt.org/page/Networking#Forwarding_Incoming_Connections</a>:</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span style="color:rgb(0,0,0);font-family:LibvirtOverpass">By default, guests that are connected via a virtual network with <forward mode='nat'/> can make any outgoing network connection they like. Incoming connections are allowed from the host, and from other guests connected to the same libvirt network, but all other incoming connections are blocked by iptables rules.</span></blockquote><div><br></div><div>Also here's another assertion from 'The virtual network driver' section in <a href="http://libvirt.org/firewall.html">http://libvirt.org/firewall.html</a>:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span style="color:rgb(0,0,0);font-family:LibvirtOverpass">type=nat </span><br style="color:rgb(0,0,0);font-family:LibvirtOverpass"><br style="color:rgb(0,0,0);font-family:LibvirtOverpass"><span style="color:rgb(0,0,0);font-family:LibvirtOverpass">Allow inbound related to an established connection. Allow outbound, but only from our expected subnet. Allow traffic between guests. Deny all other inbound. Deny all other outbound.</span></blockquote><div><br></div></div><div><br></div><div>I have three virtual networks with the following configs:<br>-----------------------------------------------------------------------------</div><div><div><font face="monospace, monospace"><network connections='1'></font></div><div><font face="monospace, monospace"> <name>default</name></font></div><div><font face="monospace, monospace"> <uuid>9c6796be-d54e-42bc-bcbe-2e4feee7154a</uuid></font></div><div><font face="monospace, monospace"> <forward mode='nat'></font></div><div><font face="monospace, monospace"> <nat></font></div><div><font face="monospace, monospace"> <port start='1024' end='65535'/></font></div><div><font face="monospace, monospace"> </nat></font></div><div><font face="monospace, monospace"> </forward></font></div><div><font face="monospace, monospace"> <bridge name='virbr0' stp='on' delay='0'/></font></div><div><font face="monospace, monospace"> <mac address='52:54:00:5a:5d:0e'/></font></div><div><font face="monospace, monospace"> <ip address='192.168.122.1' netmask='255.255.255.0'></font></div><div><font face="monospace, monospace"> <dhcp></font></div><div><font face="monospace, monospace"> <range start='192.168.122.2' end='192.168.122.254'/></font></div><div><font face="monospace, monospace"> </dhcp></font></div><div><font face="monospace, monospace"> </ip></font></div><div><font face="monospace, monospace"></network></font></div></div><div><font face="monospace, monospace"><br></font></div><div><div><font face="monospace, monospace"><network connections='1'></font></div><div><font face="monospace, monospace"> <name>outsider</name></font></div><div><font face="monospace, monospace"> <uuid>247e380a-8795-466a-b94a-5be2d05267bb</uuid></font></div><div><font face="monospace, monospace"> <forward mode='nat'></font></div><div><font face="monospace, monospace"> <nat></font></div><div><font face="monospace, monospace"> <port start='1024' end='65535'/></font></div><div><font face="monospace, monospace"> </nat></font></div><div><font face="monospace, monospace"> </forward></font></div><div><font face="monospace, monospace"> <bridge name='virbr1' stp='on' delay='0'/></font></div><div><font face="monospace, monospace"> <mac address='52:54:00:7f:a1:fb'/></font></div><div><font face="monospace, monospace"> <domain name='outsider'/></font></div><div><font face="monospace, monospace"> <ip address='192.168.100.1' netmask='255.255.255.0'></font></div><div><font face="monospace, monospace"> <dhcp></font></div><div><font face="monospace, monospace"> <range start='192.168.100.2' end='192.168.100.254'/></font></div><div><font face="monospace, monospace"> </dhcp></font></div><div><font face="monospace, monospace"> </ip></font></div><div><font face="monospace, monospace"></network></font></div></div><div><font face="monospace, monospace"><br></font></div><div><div><font face="monospace, monospace"><network connections='1'></font></div><div><font face="monospace, monospace"> <name>besider</name></font></div><div><font face="monospace, monospace"> <uuid>cc714cce-dbba-452d-b2bf-d36084dcb723</uuid></font></div><div><font face="monospace, monospace"> <forward mode='nat'></font></div><div><font face="monospace, monospace"> <nat></font></div><div><font face="monospace, monospace"> <port start='1024' end='65535'/></font></div><div><font face="monospace, monospace"> </nat></font></div><div><font face="monospace, monospace"> </forward></font></div><div><font face="monospace, monospace"> <bridge name='virbr2' stp='on' delay='0'/></font></div><div><font face="monospace, monospace"> <mac address='52:54:00:59:67:7f'/></font></div><div><font face="monospace, monospace"> <domain name='besider'/></font></div><div><font face="monospace, monospace"> <ip address='192.168.110.1' netmask='255.255.255.0'></font></div><div><font face="monospace, monospace"> <dhcp></font></div><div><font face="monospace, monospace"> <range start='192.168.110.2' end='192.168.110.254'/></font></div><div><font face="monospace, monospace"> </dhcp></font></div><div><font face="monospace, monospace"> </ip></font></div><div><font face="monospace, monospace"></network></font></div></div><div>----------------------------------------------------------------------------</div><div><br></div><div><br></div><div>Here is the output of the 'FORWARD' iptables chain rules on my host (still using firewall-cmd):</div><div>------------------------------------------------------------------------</div><div><div><font face="monospace, monospace">Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)</font></div><div><font face="monospace, monospace"> pkts bytes target prot opt in out source destination </font></div><div><font face="monospace, monospace"> 8967 14M ACCEPT all -- * virbr2 <a href="http://0.0.0.0/0">0.0.0.0/0</a> <a href="http://192.168.110.0/24">192.168.110.0/24</a> ctstate RELATED,ESTABLISHED</font></div><div><font face="monospace, monospace"> 5262 279K ACCEPT all -- virbr2 * <a href="http://192.168.110.0/24">192.168.110.0/24</a> <a href="http://0.0.0.0/0">0.0.0.0/0</a> </font></div><div><font face="monospace, monospace"> 0 0 ACCEPT all -- virbr2 virbr2 <a href="http://0.0.0.0/0">0.0.0.0/0</a> <a href="http://0.0.0.0/0">0.0.0.0/0</a> </font></div><div><font face="monospace, monospace"> 70 5832 REJECT all -- * virbr2 <a href="http://0.0.0.0/0">0.0.0.0/0</a> <a href="http://0.0.0.0/0">0.0.0.0/0</a> reject-with icmp-port-unreachable</font></div><div><font face="monospace, monospace"> 0 0 REJECT all -- virbr2 * <a href="http://0.0.0.0/0">0.0.0.0/0</a> <a href="http://0.0.0.0/0">0.0.0.0/0</a> reject-with icmp-port-unreachable</font></div><div><font face="monospace, monospace"> 8510 13M ACCEPT all -- * virbr0 <a href="http://0.0.0.0/0">0.0.0.0/0</a> <a href="http://192.168.122.0/24">192.168.122.0/24</a> ctstate RELATED,ESTABLISHED</font></div><div><font face="monospace, monospace"> 5177 275K ACCEPT all -- virbr0 * <a href="http://192.168.122.0/24">192.168.122.0/24</a> <a href="http://0.0.0.0/0">0.0.0.0/0</a> </font></div><div><font face="monospace, monospace"> 0 0 ACCEPT all -- virbr0 virbr0 <a href="http://0.0.0.0/0">0.0.0.0/0</a> <a href="http://0.0.0.0/0">0.0.0.0/0</a> </font></div><div><font face="monospace, monospace"> 61 5100 REJECT all -- * virbr0 <a href="http://0.0.0.0/0">0.0.0.0/0</a> <a href="http://0.0.0.0/0">0.0.0.0/0</a> reject-with icmp-port-unreachable</font></div><div><font face="monospace, monospace"> 0 0 REJECT all -- virbr0 * <a href="http://0.0.0.0/0">0.0.0.0/0</a> <a href="http://0.0.0.0/0">0.0.0.0/0</a> reject-with icmp-port-unreachable</font></div><div><font face="monospace, monospace"> 8612 13M ACCEPT all -- * virbr1 <a href="http://0.0.0.0/0">0.0.0.0/0</a> <a href="http://192.168.100.0/24">192.168.100.0/24</a> ctstate RELATED,ESTABLISHED</font></div><div><font face="monospace, monospace"> 5172 273K ACCEPT all -- virbr1 * <a href="http://192.168.100.0/24">192.168.100.0/24</a> <a href="http://0.0.0.0/0">0.0.0.0/0</a> </font></div><div><font face="monospace, monospace"> 0 0 ACCEPT all -- virbr1 virbr1 <a href="http://0.0.0.0/0">0.0.0.0/0</a> <a href="http://0.0.0.0/0">0.0.0.0/0</a> </font></div><div><font face="monospace, monospace"> 0 0 REJECT all -- * virbr1 <a href="http://0.0.0.0/0">0.0.0.0/0</a> <a href="http://0.0.0.0/0">0.0.0.0/0</a> reject-with icmp-port-unreachable</font></div><div><font face="monospace, monospace"> 0 0 REJECT all -- virbr1 * <a href="http://0.0.0.0/0">0.0.0.0/0</a> <a href="http://0.0.0.0/0">0.0.0.0/0</a> reject-with icmp-port-unreachable</font></div><div><font face="monospace, monospace"> 0 0 ACCEPT all -- * * <a href="http://0.0.0.0/0">0.0.0.0/0</a> <a href="http://0.0.0.0/0">0.0.0.0/0</a> ctstate RELATED,ESTABLISHED</font></div><div><font face="monospace, monospace"> 0 0 ACCEPT all -- lo * <a href="http://0.0.0.0/0">0.0.0.0/0</a> <a href="http://0.0.0.0/0">0.0.0.0/0</a> </font></div><div><font face="monospace, monospace"> 0 0 FORWARD_direct all -- * * <a href="http://0.0.0.0/0">0.0.0.0/0</a> <a href="http://0.0.0.0/0">0.0.0.0/0</a> </font></div><div><font face="monospace, monospace"> 0 0 FORWARD_IN_ZONES_SOURCE all -- * * <a href="http://0.0.0.0/0">0.0.0.0/0</a> <a href="http://0.0.0.0/0">0.0.0.0/0</a> </font></div><div><font face="monospace, monospace"> 0 0 FORWARD_IN_ZONES all -- * * <a href="http://0.0.0.0/0">0.0.0.0/0</a> <a href="http://0.0.0.0/0">0.0.0.0/0</a> </font></div><div><font face="monospace, monospace"> 0 0 FORWARD_OUT_ZONES_SOURCE all -- * * <a href="http://0.0.0.0/0">0.0.0.0/0</a> <a href="http://0.0.0.0/0">0.0.0.0/0</a> </font></div><div><font face="monospace, monospace"> 0 0 FORWARD_OUT_ZONES all -- * * <a href="http://0.0.0.0/0">0.0.0.0/0</a> <a href="http://0.0.0.0/0">0.0.0.0/0</a> </font></div><div><font face="monospace, monospace"> 0 0 DROP all -- * * <a href="http://0.0.0.0/0">0.0.0.0/0</a> <a href="http://0.0.0.0/0">0.0.0.0/0</a> ctstate INVALID</font></div><div><font face="monospace, monospace"> 0 0 REJECT all -- * * <a href="http://0.0.0.0/0">0.0.0.0/0</a> <a href="http://0.0.0.0/0">0.0.0.0/0</a> reject-with icmp-host-prohibited</font></div>--------------------------------------------------------------------------<br><br>I have a VM in each network:</div><div><a href="http://nest1.example.com">nest1.example.com</a> (virbr0) - 192.168.122.50</div><div><a href="http://nest2.example.org">nest2.example.org</a> (virbr1) - 192.168.100.100</div><div><a href="http://nest3.example.net">nest3.example.net</a> (virbr2) - 192.168.110.25<br><br>I'm quite aware the above iptables rules were added by libvirt, but I'm still managing the firewall primarily through the <b>firewall-cmd</b> command. <br><br>From what I gathered...</div><div><div>----------------------------------------------------------<br></div></div><div>nest3 can ping nest1 and nest2.<br></div><div><br></div><div><div>Nest3 ping nest1 and made a hit here:</div><div></div></div><div><div><font face="monospace, monospace"> 8967 14M ACCEPT all -- * virbr2 <a href="http://0.0.0.0/0">0.0.0.0/0</a> <a href="http://192.168.110.0/24">192.168.110.0/24</a> ctstate RELATED,ESTABLISHED</font></div><div><font face="monospace, monospace"> 5262 279K ACCEPT all -- virbr2 * <a href="http://192.168.110.0/24">192.168.110.0/24</a> <a href="http://0.0.0.0/0">0.0.0.0/0</a> </font></div></div><div><br></div><div><div><div>Nest3 ping nest2 and made a hit here:</div><div></div></div><div></div></div><div><div><font face="monospace, monospace"> 8967 14M ACCEPT all -- * virbr2 <a href="http://0.0.0.0/0">0.0.0.0/0</a> <a href="http://192.168.110.0/24">192.168.110.0/24</a> ctstate RELATED,ESTABLISHED</font></div><div><font face="monospace, monospace"> 5262 279K ACCEPT all -- virbr2 * <a href="http://192.168.110.0/24">192.168.110.0/24</a> <a href="http://0.0.0.0/0">0.0.0.0/0</a> </font></div></div><div><div>----------------------------------------------------------</div></div><div><br></div><div><br></div><div>----------------------------------------------------------<br></div><div>Nest1 can ping nest2, but cannot ping nest3.</div><div><br></div><div>Nest1 ping nest2 and made a hit here:</div><div><div><font face="monospace, monospace">8510 13M ACCEPT all -- * virbr0 <a href="http://0.0.0.0/0">0.0.0.0/0</a> <a href="http://192.168.122.0/24">192.168.122.0/24</a> ctstate RELATED,ESTABLISHED</font></div><div><font face="monospace, monospace"> 5177 275K ACCEPT all -- virbr0 * <a href="http://192.168.122.0/24">192.168.122.0/24</a> <a href="http://0.0.0.0/0">0.0.0.0/0</a> </font></div></div><div><br></div><div><div>Nest1 ping nest3 and made a hit here:</div><div></div></div><div><div><font face="monospace, monospace"> 70 5832 REJECT all -- * virbr2 <a href="http://0.0.0.0/0">0.0.0.0/0</a> <a href="http://0.0.0.0/0">0.0.0.0/0</a> reject-with icmp-port-unreachable</font></div></div><div><div>----------------------------------------------------------<br></div></div><div><br></div><div><br></div><div>----------------------------------------------------------</div><div>Nest2 cannot ping nest1 and nest3.<br></div><div><br></div><div>Nest2 ping nest1 and made a hit here:<br><div><font face="monospace, monospace"> 61 5100 REJECT all -- * virbr0 <a href="http://0.0.0.0/0">0.0.0.0/0</a> <a href="http://0.0.0.0/0">0.0.0.0/0</a> reject-with icmp-port-unreachable</font></div><div><br></div>Nest2 ping test3 and made a hit here:<br><div><font face="monospace, monospace"> 70 5832 REJECT all -- * virbr2 <a href="http://0.0.0.0/0">0.0.0.0/0</a> <a href="http://0.0.0.0/0">0.0.0.0/0</a> reject-with icmp-port-unreachable</font></div><div><div>----------------------------------------------------------</div></div><div><br></div><div><br></div><div>From my observation, I see that the order of the virtual networks in the iptables FORWARD chain makes a difference. Each chunk associated with the virtual network in the chain, consisting of five lines, is exactly as described in the aforementioned link (<a href="http://libvirt.org/firewall.html">http://libvirt.org/firewall.html</a>). The virtual network in the top chunk of the chain can communicate with virtually all networks as opposed to the network in the last chunk that is consistent with the intention of the original design.</div><div><br></div>I'm using CentOS 7.3 with libvirt 2.0. I even tried reproducing this with CentOS 6.9 as I thought it was possible the firewalld may have influenced the change, but I was still getting similar result.</div><div><br></div><div>Right now, I'm not certain if this is already a reported known bug, but I'm highly convinced this configuration is unofficially unsupported for quite a while. Can someone confirm this? This is my very first mailing list submission ever, and I apologize in advance if I couldn't figure out how to conveniently search up a similar discussion as this one in the archive. If this is in fact near impossible to implement in accordance to the intended design, then I'd like to have this confirmation publicly included in the docs.</div><div><br></div><div>Thanks,</div><div><br></div><div>Travis Johnson</div><div><br></div></div>