[libvirt] [RFC] Vhost-user backends cross-version migration support

[Michael S. Tsirkin]

On Fri, Feb 03, 2017 at 03:54:52PM +0000, Daniel P. Berrange wrote:
> On Fri, Feb 03, 2017 at 05:34:07PM +0200, Michael S. Tsirkin wrote:
> > On Fri, Feb 03, 2017 at 03:11:10PM +0100, Maxime Coquelin wrote:
> > > Hi,
> > > 
> > > On 02/01/2017 09:35 AM, Maxime Coquelin wrote:
> > > > Hi,
> > > > 
> > > >  Few months ago, Michael reported a problem about migrating VMs relying
> > > > on vhost-user between hosts supporting different backend versions:
> > > >  - Message-Id: <20161011173526-mutt-send-email-mst at kernel.org>
> > > >  - https://lists.gnu.org/archive/html/qemu-devel/2016-10/msg03026.html
> > > > 
> > > >  The goal of this thread is to draft a proposal based on the outcomes
> > > > of discussions with contributors of the different parties (DPDK/OVS
> > > > /libvirt/...).
> > > 
> > > Thanks the first feedback. It seems to converge that this is Nova's
> > > role, but not Libvirt one to manage these versions from management tool
> > > layer.
> > 
> > 
> > I think the conclusion is not that it should go up the stack.  I think
> > this will just get broken all the time.  No one understands versions and
> > stuff. Even QEMU developers get confused and break compatibility once in
> > a while.
> 
> I don't think it is really true, nor is it really specific to migration,
> it can hit any time you have a network connection that is re-opened even
> without migration. Version compatibility negotiations are an inherant part
> of any network protocol that is expected to evolve over time.
> 
> > My conclusion is that doing it from OVS side is wrong.  Migration is not
> > an OVS thing, it's a QEMU thing, and libvirt abstracts QEMU.    People
> > just want migration to work, ok? It's our job to do it, we do not really
> > need a "make things work" flag.
> > 
> > If libvirt does not want to use the vhost-user protocol (which sounds
> > reasonable, it's rather complex) how about qemu providing a small
> > utility to query the port?  We could output json or whatever.
> > 
> > This can help with MTU as well.
> > 
> > And maybe it will help with nowait support - if someone uses the utility
> > to dump backend config once, QEMU can later start the device without
> > feature queries.
> 
> I don't think it is QEMU's job to deal with external components in
> this way and don't think QEMU vhost-user should be treated as special.

It's very special because it's the only time we ship parts of
device emulation out to an external service.
So they need to be versioned together with QEMU.


> This general scenario arises any time there is a QEMU backend is talking
> to an external service/process.
> 
> For example, a virtio-serial talking to a daemon in the host. This daemon
> can support different versions of the protocol being spoken, so we have
> the same compat issue on migration. Or a traditional serial device, which
> the guest is using to talk to an external daemon the host. In a few cases
> we might know what the protocol is, but in general the data stream  is
> opaque to QEMU.

Are there any examples of people actually managing this across versions
and getting it right?

AFAIK there are cases where data stream is opaque to QEMU like -cpu host
and in every such case VM is unmigrateable.

OTOH vhost-user is not opaque to QEMU at all, and that is intentional
so we can support migration.

> QEMU should not have need to learn about every single protocol that might
> be used to communicate with some external service. This is an unbounded
> set of possibilities to deal with, some of which will not even be open
> source.
> 
> This all needs to be delegated to whatever mgmt app is responsible for
> setting up these external services, as it has the right domain knowledge
> about the applications being used, to make the policy decisions that are
> suitable for its usage scenario.
> 
> Regards,
> Daniel

I disagree it should not and in fact this is exactly what QEMU often does.
This is how in the past we were mostly able to emulate any missing
backend features in QEMU to make things transparent to guest.
E.g. it can actually support the same virtio net device with userspace and
kernel backends and migrate between them.

> -- 
> |: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
> |: http://libvirt.org              -o-             http://virt-manager.org :|
> |: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|