[libvirt-users] live migration via unix socket

[David Vossel]

On Wed, Aug 29, 2018 at 4:55 AM, Daniel P. Berrangé <berrange at redhat.com>
wrote:

> On Tue, Aug 28, 2018 at 05:07:18PM -0400, David Vossel wrote:
> > Hey,
> >
> > Over in KubeVirt we're investigating a use case where we'd like to
> perform
> > a live migration within a network namespace that does not provide
> libvirtd
> > with network access. In this scenario we would like to perform a live
> > migration by proxying the migration through a unix socket to a process in
> > another network namespace that does have network access.  That external
> > process would live on every node in the cluster and know how to correctly
> > route connections between libvirtds.
> >
> > virsh example of an attempted migration via unix socket.
> >
> > virsh migrate --copy-storage-all --p2p --live --xml domain.xml my-vm
> > qemu+unix:///system?socket=destination-host-proxy-sock
> >
> > In this example, the src libvirtd is able to establish a connection to
> the
> > destination libvirtd via the unix socket proxy. However, the
> migration-uri
> > appears to require either tcp or rdma network connection.  If I force the
> > migration-uri to be a unix socket, I receive an error [1] indicating that
> > qemu+unix is not a valid transport.
>
> qemu+unix is a syntax for libvirt's URI format.  The URI scheme for
> migration is not the same, so you can't simply plug in qemu+unix here.
>
> >
> > Technically with qemu+kvm I believe what we're attempting should be
> > possible (even though it is inefficient). Please correct me if I'm wrong.
> >
> > Is there a way to achieve this migration via unix socket functionality
> this
> > using Libvirt? Also, is there a reason why the migration uri is limited
> to
> > tcp/rdma
>
> Internally libvirt does exactly this when using its TUNNELLED live
> migration
> mode. In this QEMU is passed an anonymous UNIX socket and the data is all
> copied over the libvirtd <-> libvirtd connection and then copied again back
>

Sorry for the delayed response here, I've only just picked this task back
up again recently.

With the TUNNELLED and PEER2PEER migration flags set, Libvirt won't allow
the libvirtd <-> libvirtd connection over a unix socket.

Libvirt returns this error "Attempt to migrate guest to the same host".
The virDomainMigrateCheckNotLocal() function ensures that a peer2peer
migration won't occur when the destination is a unix socket.

Is there anyway around this? We'd like to tunnel the destination connection
through a unix socket. The other side of the unix socket is a network proxy
in a different network namespace which properly performs the remote
connection.


> to QEMU on another UNIX socket. This was done because QEMU has long had no
> ability to encrypt live migration, so tunnelling over libvirtd's own TLS
> secured connection was only secure mechanism.


> We've done work in QEMU to natively support TLS now so that we can get rid
> of this tunnelling, as this architecture decreased performance and consumed
> precious CPU memory bandwidth, which is particularly bad when libvirtd and
> QEMU were on different NUMA nodes. It is already a challenge to get live
> migration to successfully complete even with a direct network connection.
> Although QEMU can do it at the low level, we've never exposed anything
> other than direct network transports at the API level.
>
> Regards,
> Daniel
> --
> |: https://berrange.com      -o-    https://www.flickr.com/photos/
> dberrange :|
> |: https://libvirt.org         -o-
> https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/
> dberrange :|
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/libvirt-users/attachments/20180910/f065d277/attachment.htm>