[libvirt] [PATCH 00/38] Implement sparse streams for libvirt

[Michal Privoznik]

There were already some attempts to do this in the past, but
neither of them was successful. Anyway, let me just show you how
good these perform:

7.9G -rw-r--r-- 1 root root 21G Feb 15 14:04 /var/lib/libvirt/images/gentoo.qcow2

libvirt.git # time virsh vol-download --sparse /var/lib/libvirt/images/gentoo.qcow2 /mnt/floppy/gentoo.qcow2
real    0m41.627s
user    0m3.880s
sys     0m5.720s


libvirt.git # time virsh vol-download /var/lib/libvirt/images/gentoo.qcow2 /mnt/floppy/gentoo.qcow2
real    2m22.357s
user    1m20.590s
sys     0m12.510s


All the patches can be found on my github:

  https://github.com/zippy2/libvirt/tree/sparse_iohelper5

(Yes, weird name, but there were plenty of local attempts to
 implement this and I'm not throwing them away until the feature
 is merged)

Michal Privoznik (38):
  fdstreamtest: Rename tempdir
  fdstreamtest: Print more info on read failure
  fdstream: s/struct virFDStreamData */virFDStreamDataPtr/
  virFDStreamData: Turn into virObjectLockable
  virfdstream: Drop iohelper in favour of a thread
  virfdstream: Use messages instead of pipe
  iohelper: Remove unused mode
  util: Introduce virFileInData
  Introduce virStreamRecvFlags
  Implement virStreamRecvFlags to some drivers
  Introduce virStreamSkip
  Introduce virStreamHoleSize
  Introduce VIR_STREAM_RECV_STOP_AT_HOLE flag
  Introduce virStreamSparseRecvAll
  Introduce virStreamSparseSendAll
  Introduce virStreamInData
  virNetClientStreamNew: Track origin stream
  Track if stream is skippable
  RPC: Introduce virNetStreamSkip
  Introduce VIR_NET_STREAM_SKIP message type
  Teach wireshark plugin about VIR_NET_STREAM_SKIP
  daemon: Introduce virNetServerProgramSendStreamSkip
  virnetclientstream: Introduce virNetClientStreamSendSkip
  daemon: Implement VIR_NET_STREAM_SKIP handling
  virnetclientstream: Introduce virNetClientStreamHandleSkip
  remote_driver: Implement virStreamSkip
  virNetClientStreamRecvPacket: Introduce @flags argument
  Introduce virNetClientStreamHoleSize
  remote: Implement virStreamHoleSize
  virNetClientStream: Wire up VIR_NET_STREAM_SKIP
  remote_driver: Implement VIR_STREAM_RECV_STOP_AT_HOLE
  daemonStreamHandleRead: Wire up seekable stream
  daemon: Don't call virStreamInData so often
  fdstream: Implement sparse stream
  gendispatch: Introduce @sparseflag for our calls
  Introduce virStorageVol{Download,Upload}Flags
  virsh: Implement sparse stream to vol-download
  virsh: Implement sparse stream to vol-upload

 daemon/remote.c                      |   2 +-
 daemon/stream.c                      | 147 +++++-
 daemon/stream.h                      |   3 +-
 include/libvirt/libvirt-storage.h    |   9 +
 include/libvirt/libvirt-stream.h     |  86 +++-
 src/driver-stream.h                  |  23 +
 src/esx/esx_stream.c                 |  16 +-
 src/libvirt-storage.c                |   4 +-
 src/libvirt-stream.c                 | 453 ++++++++++++++++++
 src/libvirt_internal.h               |   3 +
 src/libvirt_private.syms             |   2 +
 src/libvirt_public.syms              |   9 +
 src/libvirt_remote.syms              |   3 +
 src/remote/remote_driver.c           |  89 +++-
 src/remote/remote_protocol.x         |   2 +
 src/rpc/gendispatch.pl               |  21 +-
 src/rpc/virnetclient.c               |   1 +
 src/rpc/virnetclientstream.c         | 202 +++++++-
 src/rpc/virnetclientstream.h         |  17 +-
 src/rpc/virnetprotocol.x             |  16 +-
 src/rpc/virnetserverprogram.c        |  33 ++
 src/rpc/virnetserverprogram.h        |   7 +
 src/storage/storage_driver.c         |   4 +-
 src/storage/storage_util.c           |  10 +-
 src/util/iohelper.c                  |  72 +--
 src/util/virfdstream.c               | 888 ++++++++++++++++++++++++++++-------
 src/util/virfdstream.h               |   2 +-
 src/util/virfile.c                   |  81 ++++
 src/util/virfile.h                   |   3 +
 src/virnetprotocol-structs           |   4 +
 tests/fdstreamtest.c                 |  12 +-
 tests/virfiletest.c                  | 203 ++++++++
 tools/virsh-util.c                   |  49 ++
 tools/virsh-util.h                   |  24 +
 tools/virsh-volume.c                 |  49 +-
 tools/virsh.pod                      |   6 +-
 tools/wireshark/src/packet-libvirt.c |  48 ++
 tools/wireshark/src/packet-libvirt.h |   2 +
 38 files changed, 2284 insertions(+), 321 deletions(-)

-- 
2.10.2