[libvirt] [PATCH 00/21] Support NBD for tunnelled migration

[Pavel Boldin]

The provided patchset implements NBD disk migration over a tunnelled
connection provided by libvirt.

The migration source instructs QEMU to NBD mirror drives into the provided
UNIX socket. These connections and all the data are then tunnelled to the
destination using newly introduced RPC call. The migration destination
implements a driver method that connects the tunnelled stream to the QEMU's
NBD destination.

The detailed scheme is the following:

PREPARE
 1. Migration destination starts QEMU's NBD server listening on a UNIX socket
    using the `nbd-server-add` monitor command and tells NBD to accept listed
    disks via code added to qemuMigrationStartNBDServer that calls introduced
    qemuMonitorNBDServerStartUnix monitor function.

PERFORM
 2. Migration source creates a UNIX socket that is later used as NBDs
    destination in `drive-mirror` monitor command.

    This is implemented as a call to virNetSocketNewListenUnix from
    doTunnelMigrate.

 3. Source starts IOThread that polls on the UNIX socket, accepting every
    incoming QEMU connection.
    
    This is done by adding a new pollfd in the poll(2) call in
    qemuMigrationIOFunc that calls introduced qemuNBDTunnelAcceptAndPipe
    function.

 4. The qemuNBDTunnelAcceptAndPipe function accepts the connection and creates
    two virStream's. One is `local` that is later associated with just accepted
    connection using virFDStreamOpen. Second is `remote` that is later
    tunnelled to the remote destination stream.

    The `local` stream is converted to a virFDStreamDrv stream using the
    virFDStreamOpen call on the fd returned by accept(2).

    The `remote` stream is associated with a stream on the destination in
    the way similar to used by PrepareTunnel3* function. That is, the
    virDomainMigrateOpenTunnel function called on the destination
    connection object. The virDomainMigrateOpenTunnel calls remote driver's
    handler remoteDomainMigrateOpenTunnel that makes DOMAIN_MIGRATE_OPEN_TUNNEL
    call to the destination host. The code in remoteDomainMigrateOpenTunnel
    ties passed virStream object to a virStream on the destination host via
    remoteStreamDrv driver. The remote driver handles stream's IO by tunnelling
    data through the RPC connection.

    The qemuNBDTunnelAcceptAndPipe at last assigns both streams the same event
    callback qemuMigrationPipeEvent. Its job is to track statuses of the
    streams doing IO whenever it is necessary.

 5. Source starts the drive mirroring using the qemuMigrationDriveMirror func.
    The function instructs QEMU to mirror drives to the UNIX socket that thread
    listens on.

    Since it is necessary for the mirror driving to get into the 'synchronized'
    state, where writes go to both destinations simultaneously, before
    continuing VM migration, the thread serving the connections must be
    started earlier.

 6. When the connection to a UNIX socket on the migration source is made
    the DOMAIN_MIGRATE_OPEN_TUNNEL proc is called on the migration destination.

    The handler of this code calls virDomainMigrateOpenTunnel which calls
    qemuMigrationOpenNBDTunnel by the means of qemuDomainMigrateOpenTunnel. 

    The qemuMigrationOpenNBDTunnel connects the stream linked to a source's
    stream to the NBD's UNIX socket on the migration destination side.
 
 7. The rest of the disk migration occurs semimagically: virStream* APIs tunnel
    data in both directions. This is done by qemuMigrationPipeEvent event
    callback set for both streams.


The order of the patches is roughly the following:

 *  First, the RPC machinery and remote driver's virDrvDomainMigrateOpenTunnel
    implementation are added.

 *  Then, the source-side of the protocol is implemented: code listening
    on a UNIX socket is added, DriveMirror is enhanced to instruct QEMU to
    `drive-mirror` here and starting IOThread driving the tunneling sooner.

 *  After that, the destination-side of the protocol is implemented:
    the qemuMonitorNBDServerStartUnix added and qemuMigrationStartNBDServer
    enhanced to call it. The qemuDomainMigrateOpenTunnel is implemented
    along with qemuMigrationOpenNBDTunnel that does the real job.

 *  Finally, the code blocking NBD migration for tunnelled migration is
    removed.

Pavel Boldin (21):
  rpc: add DOMAIN_MIGRATE_OPEN_TUNNEL proc
  driver: add virDrvDomainMigrateOpenTunnel
  remote_driver: introduce virRemoteClientNew
  remote_driver: add remoteDomainMigrateOpenTunnel
  domain: add virDomainMigrateOpenTunnel
  domain: add virDomainMigrateTunnelFlags
  remote: impl remoteDispatchDomainMigrateOpenTunnel
  qemu: migration: src: add nbd tunnel socket data
  qemu: migration: src: nbdtunnel unix socket
  qemu: migration: src: qemu `drive-mirror` to UNIX
  qemu: migration: src: qemuSock for running thread
  qemu: migration: src: add NBD unixSock to iothread
  qemu: migration: src: qemuNBDTunnelAcceptAndPipe
  qemu: migration: src: stream piping
  qemu: monitor: add qemuMonitorNBDServerStartUnix
  qemu: migration: dest: nbd-server to UNIX sock
  qemu: migration: dest: qemuMigrationOpenTunnel
  qemu: driver: add qemuDomainMigrateOpenTunnel
  qemu: migration: dest: qemuMigrationOpenNBDTunnel
  qemu: migration: allow NBD tunneling migration
  apparmor: fix tunnelmigrate permissions

 daemon/remote.c                  |  50 ++++
 docs/apibuild.py                 |   1 +
 docs/hvsupport.pl                |   1 +
 include/libvirt/libvirt-domain.h |   3 +
 src/driver-hypervisor.h          |   8 +
 src/libvirt-domain.c             |  43 ++++
 src/libvirt_internal.h           |   6 +
 src/libvirt_private.syms         |   1 +
 src/qemu/qemu_driver.c           |  24 ++
 src/qemu/qemu_migration.c        | 495 +++++++++++++++++++++++++++++++++------
 src/qemu/qemu_migration.h        |   6 +
 src/qemu/qemu_monitor.c          |  12 +
 src/qemu/qemu_monitor.h          |   2 +
 src/qemu/qemu_monitor_json.c     |  35 +++
 src/qemu/qemu_monitor_json.h     |   2 +
 src/remote/remote_driver.c       |  91 +++++--
 src/remote/remote_protocol.x     |  19 +-
 src/remote_protocol-structs      |   8 +
 src/security/virt-aa-helper.c    |   4 +-
 19 files changed, 719 insertions(+), 92 deletions(-)

-- 
1.9.1