[libvirt-users] macvtap and tagged VLANs to the VM

Wed Jan 2 16:34:30 UTC 2019

On 12/16/18 4:59 PM, Marc Haber wrote:
> Hi,
> 
> I would like to run a network firewall as a VM on a KVM host. There are
> ~ 25 VLANs delivered to the KVM host on three dedicated links, no LACP
> or other things. I have the VLANs 100-180 on the host's enp1s0, the VLANs
> 200-280 on the host's enp2s0 and the VLANs 300-380 on the host's enp3s0.
> 
> To save myself from configuring all VLANs on the KVM host, I'd like to
> hand the entire ethernet link to the VM and to have the VLAN interfaces
> there. Using classical Linux bridges (brctl), things work fine.

When I asked the person I go to with questions about macvtap (because he 
knows the internals), his response was "if a Linux host bridge works, 
then he should use that". In other words, he was skeptical that what you 
want to do could be made to work with macvtap.

Is there a specific reason you need to use macvtap than a Linux host bridge?

> 
> They don't when I try macvlan:
> 
> On the host:
> 4: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
>      link/ether 00:0d:b9:34:2a:fe brd ff:ff:ff:ff:ff:ff promiscuity 1 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
> 5: unt382 at enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
>      link/ether 00:0d:b9:34:2a:fe brd ff:ff:ff:ff:ff:ff promiscuity 0
>      vlan protocol 802.1Q id 382 <REORDER_HDR> addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
> 15: macvtap3 at enp3s0: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 500
>      link/ether 52:54:00:bf:bb:ab brd ff:ff:ff:ff:ff:ff promiscuity 0
>      macvtap mode bridge addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
> 
> 4: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
>      link/ether 00:0d:b9:34:2a:fe brd ff:ff:ff:ff:ff:ff
>      inet6 fe80::20d:b9ff:fe34:2afe/64 scope link
>         valid_lft forever preferred_lft forever
> 5: unt382 at enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
>      link/ether 00:0d:b9:34:2a:fe brd ff:ff:ff:ff:ff:ff
>      inet6 fe80::20d:b9ff:fe34:2afe/64 scope link
>         valid_lft forever preferred_lft forever
> 15: macvtap3 at enp3s0: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 500
>      link/ether 52:54:00:bf:bb:ab brd ff:ff:ff:ff:ff:ff
>      inet6 fe80::5054:ff:febf:bbab/64 scope link
>         valid_lft forever preferred_lft forever
> 
> 
> In the XML:
>      <interface type='direct'>
>        <mac address='52:54:00:bf:bb:ab'/>
>        <source dev='enp3s0' mode='bridge'/>
>        <target dev='macvtap3'/>
>        <model type='virtio'/>
>        <alias name='net0'/>
>        <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
>      </interface>
> 
> And in the VM:
> root at grml ~ # ip -d link show
> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1
>      link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
>      link/ether 52:54:00:bf:bb:ab brd ff:ff:ff:ff:ff:ff promiscuity 0 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
> 3: vlan0 at eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
>      link/ether 52:54:00:bf:bb:ab brd ff:ff:ff:ff:ff:ff promiscuity 0
>      vlan protocol 802.1Q id 382 <REORDER_HDR> addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
> root at grml ~ # ip a
> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
>      link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
>      inet 127.0.0.1/8 scope host lo
>         valid_lft forever preferred_lft forever
>      inet6 ::1/128 scope host
>         valid_lft forever preferred_lft forever
> 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
>      link/ether 52:54:00:bf:bb:ab brd ff:ff:ff:ff:ff:ff
>      inet6 fe80::5054:ff:febf:bbab/64 scope link
>         valid_lft forever preferred_lft forever
> 3: vlan0 at eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
>      link/ether 52:54:00:bf:bb:ab brd ff:ff:ff:ff:ff:ff
>      inet 192.168.252.220/24 brd 192.168.252.255 scope global vlan0
>         valid_lft forever preferred_lft forever
>      inet6 fe80::5054:ff:febf:bbab/64 scope link
>         valid_lft forever preferred_lft forever
> root at grml ~ #
> 
> I then ping from the VM to 192.168.252.241, which is a differnt host on
> the network, neither the VM host the VM is running on nor another VM on
> the same host. That should rule out the connectivity issues that a
> macvtap interface has, right? On the VM, I see ARP requests going out,
> but no answers come in.
> 
> On the pinged host, I see:
> 22:50:23.881163 52:54:00:bf:bb:ab > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.252.241 tell 192.168.252.220, length 46
> 22:50:23.881242 52:54:00:95:df:a6 > 52:54:00:bf:bb:ab, ethertype ARP (0x0806), length 42: Reply 192.168.252.241 is-at 52:54:00:95:df:a6, length 28
> 
> So, the packets going out from my VM are correctly delivered to the
> target, the target replies, but the replies never make it back to the
> VM.
> 
> Do I see correctly that tcpdump on the VM host won't give accurate
> readings since macvtap will divert the frame before tcpdump will see it?
> 
> On the other hand, a VM directly configured to the host's unt382
> interface works fine:
>      <interface type='direct'>
>        <mac address='52:54:00:cb:ed:34'/>
>        <source dev='unt382' mode='bridge'/>
>        <model type='virtio'/>
>        <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
>      </interface>
> I would however like to avoid having 25 interface stanzas in my XML.
> 
> I would appeciate any ideas to solve this issue. I know this is most
> probably not a libvirt issue, but this list is about the only place that
> comes to my mind where people knowledgeable about those complex network
> stuff might hang around. If there is a better place to ask, I am open
> for suggestion. Please pardon my intrusion.
> 
> Greetings
> Marc
> 