[libvirt-users] macvtap and tagged VLANs to the VM

Sun Dec 16 21:59:22 UTC 2018

Hi,

I would like to run a network firewall as a VM on a KVM host. There are
~ 25 VLANs delivered to the KVM host on three dedicated links, no LACP
or other things. I have the VLANs 100-180 on the host's enp1s0, the VLANs
200-280 on the host's enp2s0 and the VLANs 300-380 on the host's enp3s0.

To save myself from configuring all VLANs on the KVM host, I'd like to
hand the entire ethernet link to the VM and to have the VLAN interfaces
there. Using classical Linux bridges (brctl), things work fine.

They don't when I try macvlan:

On the host:
4: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether 00:0d:b9:34:2a:fe brd ff:ff:ff:ff:ff:ff promiscuity 1 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 
5: unt382 at enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 00:0d:b9:34:2a:fe brd ff:ff:ff:ff:ff:ff promiscuity 0 
    vlan protocol 802.1Q id 382 <REORDER_HDR> addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 
15: macvtap3 at enp3s0: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 500
    link/ether 52:54:00:bf:bb:ab brd ff:ff:ff:ff:ff:ff promiscuity 0 
    macvtap mode bridge addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535

4: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0d:b9:34:2a:fe brd ff:ff:ff:ff:ff:ff
    inet6 fe80::20d:b9ff:fe34:2afe/64 scope link 
       valid_lft forever preferred_lft forever
5: unt382 at enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:0d:b9:34:2a:fe brd ff:ff:ff:ff:ff:ff
    inet6 fe80::20d:b9ff:fe34:2afe/64 scope link 
       valid_lft forever preferred_lft forever
15: macvtap3 at enp3s0: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
    link/ether 52:54:00:bf:bb:ab brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:febf:bbab/64 scope link 
       valid_lft forever preferred_lft forever

In the XML:
    <interface type='direct'>
      <mac address='52:54:00:bf:bb:ab'/>
      <source dev='enp3s0' mode='bridge'/>
      <target dev='macvtap3'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

And in the VM:
root at grml ~ # ip -d link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether 52:54:00:bf:bb:ab brd ff:ff:ff:ff:ff:ff promiscuity 0 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 
3: vlan0 at eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 52:54:00:bf:bb:ab brd ff:ff:ff:ff:ff:ff promiscuity 0 
    vlan protocol 802.1Q id 382 <REORDER_HDR> addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 
root at grml ~ # ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:bf:bb:ab brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:febf:bbab/64 scope link 
       valid_lft forever preferred_lft forever
3: vlan0 at eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 52:54:00:bf:bb:ab brd ff:ff:ff:ff:ff:ff
    inet 192.168.252.220/24 brd 192.168.252.255 scope global vlan0
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:febf:bbab/64 scope link 
       valid_lft forever preferred_lft forever
root at grml ~ # 

I then ping from the VM to 192.168.252.241, which is a differnt host on
the network, neither the VM host the VM is running on nor another VM on
the same host. That should rule out the connectivity issues that a
macvtap interface has, right? On the VM, I see ARP requests going out,
but no answers come in.

On the pinged host, I see:
22:50:23.881163 52:54:00:bf:bb:ab > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.252.241 tell 192.168.252.220, length 46
22:50:23.881242 52:54:00:95:df:a6 > 52:54:00:bf:bb:ab, ethertype ARP (0x0806), length 42: Reply 192.168.252.241 is-at 52:54:00:95:df:a6, length 28

So, the packets going out from my VM are correctly delivered to the
target, the target replies, but the replies never make it back to the
VM.

Do I see correctly that tcpdump on the VM host won't give accurate
readings since macvtap will divert the frame before tcpdump will see it?

On the other hand, a VM directly configured to the host's unt382
interface works fine:
    <interface type='direct'>
      <mac address='52:54:00:cb:ed:34'/>
      <source dev='unt382' mode='bridge'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
I would however like to avoid having 25 interface stanzas in my XML.

I would appeciate any ideas to solve this issue. I know this is most
probably not a libvirt issue, but this list is about the only place that
comes to my mind where people knowledgeable about those complex network
stuff might hang around. If there is a better place to ask, I am open
for suggestion. Please pardon my intrusion.

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421