[libvirt] [PATCH RFC 00/48] Sparse streams

[Michal Privoznik]

So, after some time spent working on this, I think I finally have
something that works. Finally, we have a formatted messages
between iohelper and the daemon. BUT, it there's a corner case
because where this does not work 100%. If there's a hole at EOF,
it is either not transferred to the other side or not fully
seeked. Therefore, hole at EOF is not in the copy. I guess it is
some silly mistake somewhere. Anyway, any comments appreciated.

v2 can be found here:

https://www.redhat.com/archives/libvir-list/2016-May/msg01691.html

Michal Privoznik (48):
  util: Introduce virFileInData
  src: Move iohelper out from utils/ to a separate dir
  fdstream: s/struct virFDStreamData */virFDStreamDataPtr/
  virFDStreamData: Turn into virObjectLockable
  Introduce virStreamRecvFlags
  Implement virStreamRecvFlags to some drivers
  Introduce virStreamSkip
  Introduce virStreamHoleSize
  Introduce VIR_STREAM_RECV_STOP_AT_HOLE flag
  Introduce virStreamSparseRecvAll
  Introduce virStreamSparseSendAll
  Introduce virStreamInData
  virNetClientStreamNew: Track origin stream
  Track if stream is skippable
  RPC: Introduce virNetStreamSkip
  Introduce VIR_NET_STREAM_SKIP message type
  Teach wireshark plugin about VIR_NET_STREAM_SKIP
  daemon: Introduce virNetServerProgramSendStreamSkip
  virnetclientstream: Introduce virNetClientStreamSendSkip
  daemon: Implement VIR_NET_STREAM_SKIP handling
  virnetclientstream: Introduce virNetClientStreamHandleSkip
  remote_driver: Implement virStreamSkip
  virNetClientStreamRecvPacket: Introduce @flags argument
  Introduce virNetClientStreamHoleSize
  remote: Implement virStreamHoleSize
  virNetClientStream: Wire up VIR_NET_STREAM_SKIP
  remote_driver: Implement VIR_STREAM_RECV_STOP_AT_HOLE
  daemonStreamHandleRead: Wire up seekable stream
  fdstream: Implement seek
  gendispatch: Introduce @sparseflag for our calls
  Introduce virStorageVol{Download,Upload}Flags
  virsh: Implement sparse stream to vol-download
  virsh: Implement sparse stream to vol-upload
  daemon: Don't call virStreamInData so often
  storage: Enable sparse streams for virStorageVol{Download,Upload}
  fdstream: Track formatted message
  fdstream: Handle formatted messages separately
  iohelper: Introduce iohelper_message
  iohelper_message: Introduce API stubs
  iohelper_message: Implement formatted read
  iohelper_message: Implement formatted write
  tests: Introduce iohelpermessagetest
  iohelpermessagetest: test non-blocking read & write
  iohelper_message: Add support for sparse streams
  iohelper: Move runIO to runIOBasic
  iohelper: Introduce setupFDs
  iohelper: Teach command line 'sparse' argument
  iohelper: Wire up formatted messages

 daemon/remote.c                      |   2 +-
 daemon/stream.c                      | 152 +++++++++++-
 daemon/stream.h                      |   3 +-
 include/libvirt/libvirt-storage.h    |   9 +
 include/libvirt/libvirt-stream.h     |  86 ++++++-
 po/POTFILES.in                       |   2 +-
 src/Makefile.am                      |  33 ++-
 src/driver-stream.h                  |  23 ++
 src/esx/esx_stream.c                 |  16 +-
 src/fdstream.c                       | 267 +++++++++++++++-----
 src/fdstream.h                       |   3 +-
 src/{util => iohelper}/iohelper.c    | 234 +++++++++++++++---
 src/iohelper/iohelper_message.c      | 429 ++++++++++++++++++++++++++++++++
 src/iohelper/iohelper_message.h      |  53 ++++
 src/libvirt-storage.c                |   4 +-
 src/libvirt-stream.c                 | 455 ++++++++++++++++++++++++++++++++++
 src/libvirt_internal.h               |   3 +
 src/libvirt_private.syms             |   2 +
 src/libvirt_public.syms              |   8 +-
 src/libvirt_remote.syms              |   3 +
 src/remote/remote_driver.c           |  89 ++++++-
 src/remote/remote_protocol.x         |   2 +
 src/rpc/gendispatch.pl               |  21 +-
 src/rpc/virnetclient.c               |   1 +
 src/rpc/virnetclientstream.c         | 203 ++++++++++++++-
 src/rpc/virnetclientstream.h         |  17 +-
 src/rpc/virnetprotocol.x             |  16 +-
 src/rpc/virnetserverprogram.c        |  33 +++
 src/rpc/virnetserverprogram.h        |   7 +
 src/storage/storage_backend.c        |  12 +-
 src/storage/storage_driver.c         |   4 +-
 src/util/virfile.c                   |  70 ++++++
 src/util/virfile.h                   |   4 +
 src/virnetprotocol-structs           |   4 +
 tests/Makefile.am                    |  10 +-
 tests/iohelpermessagetest.c          | 466 +++++++++++++++++++++++++++++++++++
 tools/virsh-volume.c                 |  49 ++--
 tools/virsh.c                        |  36 +++
 tools/virsh.h                        |  17 ++
 tools/virsh.pod                      |   6 +-
 tools/wireshark/src/packet-libvirt.c |  40 +++
 tools/wireshark/src/packet-libvirt.h |   2 +
 42 files changed, 2716 insertions(+), 180 deletions(-)
 rename src/{util => iohelper}/iohelper.c (58%)
 create mode 100644 src/iohelper/iohelper_message.c
 create mode 100644 src/iohelper/iohelper_message.h
 create mode 100644 tests/iohelpermessagetest.c

-- 
2.8.4