[libvirt] Re: Supporting vhost-net and macvtap in libvirt for QEMU

[Arnd Bergmann]

On Thursday 17 December 2009, Anthony Liguori wrote:

> There are two modes worth supporting for vhost-net in libvirt.  The 
> first mode is where vhost-net backs to a tun/tap device.  This is 
> behaves in very much the same way that -net tap behaves in qemu today.  
> Basically, the difference is that the virtio backend is in the kernel 
> instead of in qemu so there should be some performance improvement.
> 
> Current, libvirt invokes qemu with -net tap,fd=X where X is an already 
> open fd to a tun/tap device.  I suspect that after we merge vhost-net, 
> libvirt could support vhost-net in this mode by just doing -net 
> vhost,fd=X.  I think the only real question for libvirt is whether to 
> provide a user visible switch to use vhost or to just always use vhost 
> when it's available and it makes sense.  Personally, I think the later 
> makes sense.

I think it should be treated like any other option where we have kernel
support to make something "go fast", e.g. kvm, or the in-kernel interrupt
processing. If we don't enable it by default when it's there, I would
prefer to have an '--enable-vhost' option to replacing the '-net tap'
option with '-net vhost', because that would be easier to integrate
with existing scripts.
 
> The more interesting invocation of vhost-net though is one where the 
> vhost-net device backs directly to a physical network card.  In this 
> mode, vhost should get considerably better performance than the current 
> implementation.  I don't know the syntax yet, but I think it's 
> reasonable to assume that it will look something like -net 
> tap,dev=eth0.   The effect will be that eth0 is dedicated to the guest.
> 
> On most modern systems, there is a small number of network devices so 
> this model is not all that useful except when dealing with SR-IOV 
> adapters.  In that case, each physical device can be exposed as many 
> virtual devices (VFs).  There are a few restrictions here though.  The 
> biggest is that currently, you can only change the number of VFs by 
> reloading a kernel module so it's really a parameter that must be set at 
> startup time.

I like to think of this way of using SR-IOV (VMDq actually) as a way to
do macvlan with hardware support. Unfortunately, it does work like
this yet, but the way I would like to do it is:

* use 'ip link ... type macvlan' as the configuration frontend for this mode
* as long as there are VFs, PFs or queue pairs available in hardware, use them
* When you run out of PCI functions, register additional unicast MAC addresses
  with the hardware, as macvlan does today
* As the final fallback, when we run out of unicast MAC addresses in the
   NIC, put it into promiscuous mode. Again, macvlan handles this fine today.

Right now, if you want to use a VF, you have to set up either raw socket,
because you can't add a tun/tap device to the interface without a bridge,
which would defeat the whole purpose of doing this.

Macvtap should eventually solve this, but only after VMDq is integrated
with macvlan.

> I think there are a few ways libvirt could support vhost-net in this 
> second mode.  The simplest would be to introduce a new tag similar to 
> <source network='br0'>.  In fact, if you probed the device type for the 
> network parameter, you could probably do something like <source 
> network='eth0'> and have it Just Work.

Right. The first option (source network='br0) is not so ideal because it
assumes that you run a bridge, which you typically don't want in this
mode, because those devices have the bridge in hardware (or in
macvlan for the software case)

	Arnd