[libvirt] [PATCH 00/21] Support NBD for tunnelled migration

Wed Dec 2 14:16:03 UTC 2015

Ping.

May I have your attention guys?

Pavel

On Wed, Nov 18, 2015 at 8:12 PM, Pavel Boldin <pboldin at mirantis.com> wrote:

> The provided patchset implements NBD disk migration over a tunnelled
> connection provided by libvirt.
>
> The migration source instructs QEMU to NBD mirror drives into the provided
> UNIX socket. These connections and all the data are then tunnelled to the
> destination using newly introduced RPC call. The migration destination
> implements a driver method that connects the tunnelled stream to the QEMU's
> NBD destination.
>
> The detailed scheme is the following:
>
> PREPARE
>  1. Migration destination starts QEMU's NBD server listening on a UNIX
> socket
>     using the `nbd-server-add` monitor command and tells NBD to accept
> listed
>     disks via code added to qemuMigrationStartNBDServer that calls
> introduced
>     qemuMonitorNBDServerStartUnix monitor function.
>
> PERFORM
>  2. Migration source creates a UNIX socket that is later used as NBDs
>     destination in `drive-mirror` monitor command.
>
>     This is implemented as a call to virNetSocketNewListenUnix from
>     doTunnelMigrate.
>
>  3. Source starts IOThread that polls on the UNIX socket, accepting every
>     incoming QEMU connection.
>
>     This is done by adding a new pollfd in the poll(2) call in
>     qemuMigrationIOFunc that calls introduced qemuNBDTunnelAcceptAndPipe
>     function.
>
>  4. The qemuNBDTunnelAcceptAndPipe function accepts the connection and
> creates
>     two virStream's. One is `local` that is later associated with just
> accepted
>     connection using virFDStreamOpen. Second is `remote` that is later
>     tunnelled to the remote destination stream.
>
>     The `local` stream is converted to a virFDStreamDrv stream using the
>     virFDStreamOpen call on the fd returned by accept(2).
>
>     The `remote` stream is associated with a stream on the destination in
>     the way similar to used by PrepareTunnel3* function. That is, the
>     virDomainMigrateOpenTunnel function called on the destination
>     connection object. The virDomainMigrateOpenTunnel calls remote driver's
>     handler remoteDomainMigrateOpenTunnel that makes
> DOMAIN_MIGRATE_OPEN_TUNNEL
>     call to the destination host. The code in remoteDomainMigrateOpenTunnel
>     ties passed virStream object to a virStream on the destination host via
>     remoteStreamDrv driver. The remote driver handles stream's IO by
> tunnelling
>     data through the RPC connection.
>
>     The qemuNBDTunnelAcceptAndPipe at last assigns both streams the same
> event
>     callback qemuMigrationPipeEvent. Its job is to track statuses of the
>     streams doing IO whenever it is necessary.
>
>  5. Source starts the drive mirroring using the qemuMigrationDriveMirror
> func.
>     The function instructs QEMU to mirror drives to the UNIX socket that
> thread
>     listens on.
>
>     Since it is necessary for the mirror driving to get into the
> 'synchronized'
>     state, where writes go to both destinations simultaneously, before
>     continuing VM migration, the thread serving the connections must be
>     started earlier.
>
>  6. When the connection to a UNIX socket on the migration source is made
>     the DOMAIN_MIGRATE_OPEN_TUNNEL proc is called on the migration
> destination.
>
>     The handler of this code calls virDomainMigrateOpenTunnel which calls
>     qemuMigrationOpenNBDTunnel by the means of qemuDomainMigrateOpenTunnel.
>
>     The qemuMigrationOpenNBDTunnel connects the stream linked to a source's
>     stream to the NBD's UNIX socket on the migration destination side.
>
>  7. The rest of the disk migration occurs semimagically: virStream* APIs
> tunnel
>     data in both directions. This is done by qemuMigrationPipeEvent event
>     callback set for both streams.
>
>
> The order of the patches is roughly the following:
>
>  *  First, the RPC machinery and remote driver's
> virDrvDomainMigrateOpenTunnel
>     implementation are added.
>
>  *  Then, the source-side of the protocol is implemented: code listening
>     on a UNIX socket is added, DriveMirror is enhanced to instruct QEMU to
>     `drive-mirror` here and starting IOThread driving the tunneling sooner.
>
>  *  After that, the destination-side of the protocol is implemented:
>     the qemuMonitorNBDServerStartUnix added and qemuMigrationStartNBDServer
>     enhanced to call it. The qemuDomainMigrateOpenTunnel is implemented
>     along with qemuMigrationOpenNBDTunnel that does the real job.
>
>  *  Finally, the code blocking NBD migration for tunnelled migration is
>     removed.
>
> Pavel Boldin (21):
>   rpc: add DOMAIN_MIGRATE_OPEN_TUNNEL proc
>   driver: add virDrvDomainMigrateOpenTunnel
>   remote_driver: introduce virRemoteClientNew
>   remote_driver: add remoteDomainMigrateOpenTunnel
>   domain: add virDomainMigrateOpenTunnel
>   domain: add virDomainMigrateTunnelFlags
>   remote: impl remoteDispatchDomainMigrateOpenTunnel
>   qemu: migration: src: add nbd tunnel socket data
>   qemu: migration: src: nbdtunnel unix socket
>   qemu: migration: src: qemu `drive-mirror` to UNIX
>   qemu: migration: src: qemuSock for running thread
>   qemu: migration: src: add NBD unixSock to iothread
>   qemu: migration: src: qemuNBDTunnelAcceptAndPipe
>   qemu: migration: src: stream piping
>   qemu: monitor: add qemuMonitorNBDServerStartUnix
>   qemu: migration: dest: nbd-server to UNIX sock
>   qemu: migration: dest: qemuMigrationOpenTunnel
>   qemu: driver: add qemuDomainMigrateOpenTunnel
>   qemu: migration: dest: qemuMigrationOpenNBDTunnel
>   qemu: migration: allow NBD tunneling migration
>   apparmor: fix tunnelmigrate permissions
>
>  daemon/remote.c                  |  50 ++++
>  docs/apibuild.py                 |   1 +
>  docs/hvsupport.pl                |   1 +
>  include/libvirt/libvirt-domain.h |   3 +
>  src/driver-hypervisor.h          |   8 +
>  src/libvirt-domain.c             |  43 ++++
>  src/libvirt_internal.h           |   6 +
>  src/libvirt_private.syms         |   1 +
>  src/qemu/qemu_driver.c           |  24 ++
>  src/qemu/qemu_migration.c        | 495
> +++++++++++++++++++++++++++++++++------
>  src/qemu/qemu_migration.h        |   6 +
>  src/qemu/qemu_monitor.c          |  12 +
>  src/qemu/qemu_monitor.h          |   2 +
>  src/qemu/qemu_monitor_json.c     |  35 +++
>  src/qemu/qemu_monitor_json.h     |   2 +
>  src/remote/remote_driver.c       |  91 +++++--
>  src/remote/remote_protocol.x     |  19 +-
>  src/remote_protocol-structs      |   8 +
>  src/security/virt-aa-helper.c    |   4 +-
>  19 files changed, 719 insertions(+), 92 deletions(-)
>
> --
> 1.9.1
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/libvir-list/attachments/20151202/48808060/attachment-0001.htm>