[libvirt] [PATCH 0/6] [RFC] Add disk streaming support to libvirt

[Adam Litke]

( Those on Cc: sorry for any duplicated messages.  git was configured to use
the wrong smtp relay )

I've been working with Anthony Liguori and Stefan Hajnoczi to enable data
streaming to copy-on-read disk images in qemu.  This work is working its way
through review and I expect it to be upstream soon as part of the support for
the new QED disk image format.

Disk streaming is extremely useful when provisioning domains from a central
repository of template images.  Currently the domain must be provisioned by
either: 1) copying the template image to local storage before the VM can be
started or, 2) creating a qcow2 image that backs to a base image in the remote
repository.  Option 1 can introduce a significant delay when provisioning large
disks.  Option 2 introduces a permanent dependency on a remote service,
increased network load to satisfy disk reads.

Device streaming provides the "instant-on" benefits of option 2 without
introducing a permanent dependency to the image repository.  Once the VM is
started, the contents of the disk can be streamed to the local image in
parallel.  Once streaming is finished, the domain has a complete and coherent
copy of the image and no longer depends on the central image repository.

Qemu will support two streaming modes: full device and single sector.  Full
device streaming is the easiest to use because one command will cause the whole
device to be streamed as fast as possible.  Single sector mode can be used if
one wants to throttle streaming to reduce I/O pressure.  In this mode, a
management tool issues individual commands to stream single sectors.

To enable this support in libvirt, I propose the following API...

virDomainStreamDisk() will start or stop a full device stream or stream a
single sector of a device.  The behavior is controlled by setting
virDomainStreamDiskFlags.  When either starting or stopping a full device
stream, the return value is either 0 or -1 to indicate whether the operation
succeeded.  For a single sector stream, a device offset is returned (or -1 on
failure).  This value can be used to continue streaming with a subsequent call
to virDomainStreamDisk().

virDomainStreamDiskInfo() returns information about active full device streams
(the device alias, current streaming position, and total size).

While streaming can be used to enable live block migration, this API addresses
the direct streaming use case.  If we can agree that the above use case is
compelling on its own, I would prefer to work on the block migration workflow
immediately after this series.  In that case, we just need to be careful not to
paint ourselves into a corner with respect to migration, but specifics about
that feature can be discussed later.  Does this seem reasonable?

Adam Litke (6):
  Add new API virDomainStreamDisk[Info] to header and drivers
  virDomainStreamDisk: Add public symbols to libvirt API
  Implement disk streaming in the qemu driver
  Add disk streaming support to the remote driver
  Add new disk streaming commands to virsh
  python: Add python bindings for virDomainStreamDisk[Info]

 daemon/remote.c                     |   96 +++++++++++++++++++++
 daemon/remote_dispatch_args.h       |    2 +
 daemon/remote_dispatch_prototypes.h |   16 ++++
 daemon/remote_dispatch_ret.h        |    2 +
 daemon/remote_dispatch_table.h      |   10 ++
 include/libvirt/libvirt.h.in        |   34 ++++++++
 python/generator.py                 |    1 +
 python/libvirt-override-api.xml     |    5 +
 python/libvirt-override.c           |   46 ++++++++++
 src/driver.h                        |   11 +++
 src/esx/esx_driver.c                |    2 +
 src/libvirt.c                       |  121 +++++++++++++++++++++++++++
 src/libvirt_public.syms             |    6 ++
 src/lxc/lxc_driver.c                |    2 +
 src/opennebula/one_driver.c         |    2 +
 src/openvz/openvz_driver.c          |    2 +
 src/phyp/phyp_driver.c              |    2 +
 src/qemu/qemu_driver.c              |   75 +++++++++++++++++
 src/qemu/qemu_monitor.c             |   41 +++++++++
 src/qemu/qemu_monitor.h             |    6 ++
 src/qemu/qemu_monitor_json.c        |  104 +++++++++++++++++++++++
 src/qemu/qemu_monitor_json.h        |    7 ++
 src/qemu/qemu_monitor_text.c        |  156 ++++++++++++++++++++++++++++++++++
 src/qemu/qemu_monitor_text.h        |    8 ++
 src/remote/remote_driver.c          |   85 +++++++++++++++++++
 src/remote/remote_protocol.c        |   63 ++++++++++++++
 src/remote/remote_protocol.h        |   51 +++++++++++
 src/remote/remote_protocol.x        |   37 ++++++++-
 src/test/test_driver.c              |    2 +
 src/uml/uml_driver.c                |    2 +
 src/vbox/vbox_tmpl.c                |    2 +
 src/xen/xen_driver.c                |    2 +
 tools/virsh.c                       |  157 +++++++++++++++++++++++++++++++++++
 33 files changed, 1157 insertions(+), 1 deletions(-)

-- 
1.7.3.2.164.g6f10c