[libvirt-users] KVM-Docker-Networking using TAP and MACVLAN

[Lars Lindstrom]

On 3/14/19 4:06 PM, Martin Kletzander wrote:
> I cannot try this right now, but I would try something like this:
>
> ip link add dev veth-vm type veth peer name veth-cont
>
> and then put veth-vm in the VM (type='direct' would work, but I can
> imagine
> type='ethernet' might be even faster) and start the containers with
> macvtap
> using veth-cont.


I really appreciate your effort. The problem is in fact that I
misunderstood the manual. I thought that VETH requires two existing
devices, but in fact it creates two devices being connected (this
virtual link exists purely in the kernel, so performance should be fine).

As there is still no connectivity I reduced the setup to the bare
minimum, I kickstarted the server, installed QEMU and defined two
Debian-based KVM with a 'direct' device in 'vepa' mode and a VETH
between them. When started, both KVM create a MACVTAP assigned to each
end of the VETH. Both KVM can ping each other (without any additional
route configured) and the VETH is LOWER_UP and UP even when no KVM is
running.

I then replaced one Debian-based KVM with pfSense and both KVM can still
ping each other.

I then created a MACVLAN docker network with the VETH as parent and
replaced the second Debian-based KVM with a Docker container using this
network. I am not quite sure why - but there is connectivity now. There
must have been some configuration issue on the server that was resolved
with kickstarting it.

Using type ETHERNET and the VETH as target did not work, "Unable to
create tap device veth1: Invalid argument". I then removed the VETH and
kept the ETHERNET configuration setting, which caused a TAP device
(according to error message) being created when the KVM is started. To
my confusion, this TAP device can be actually used as a parent for the
Docker MACVLAN network while still having connectivity! The downside is
that there is now an order dependency; the Docker network is unusable
when the KVM is shut down. In addition, when the KVM is shut down while
the container is still running the container must be restarted to get
networking going again.

So, there are two possible ways to achieve connectivity now
(VETH/VEPA/PASSTHROUGH and ETHERNET/TAP). Unfortunately there is a
'but'. I then added another container and even though the KVM device has
been configured VEPA, the containers are still able to contact each
other, so there is no isolation. I assume this is because just the 'KVM
end' of the VETH is in VEPA mode, whereas the 'Docker end' of the VETH
is in BRIDGE mode. Unfortunately for the ETHERNET/TAP way no mode can be
configured in the KVM domain (I would assume because no MACVTAP is
involved).

The problem is that I cannot seem to figure out how to configure the
Docker network to use VEPA mode.


br Lars