[Libguestfs] Support for NBD block status in nbdkit & qemu NBD client

[Eric Blake]

On 3/7/19 9:19 AM, Richard W.M. Jones wrote:
> Hi Eric:
> 
> Here's the problem: We have a VMware VDDK file or snapshot.  The
> virtual size of the snapshot is generally much larger than what's
> allocated.  Copying the snapshot using:
> 
>   nbdkit vddk file=foo.vmdk --run 'qemu-img convert $nbd out'
> 
> is therefore much slower than it could be.  (In the real world version
> of this we're actually copying the source over the network, but the
> command line for that is too complicated to show in a simple example).
> 
> VDDK has an API for querying allocated blocks, see
> ‘VixDiskLib_QueryAllocatedBlocks’ in
> https://vdc-download.vmware.com/vmwb-repository/dcr-public/8f96698a-0e7b-4d67-bb6c-d18a1d101540/ef536a47-27cd-481a-90ef-76b38e75353c/vsphere-vddk-671-programming-guide.pdf
> 
> It's awkward to use but Martin (CC'd) has played with it and it does
> seem to work.
> 
> The question is if we add Structured Replies + Block Status support to
> nbdkit & implement it for the VDDK plugin, will ‘qemu-img convert’
> above be able to use that to optimize the copy?  My brief reading of
> the qemu sources suggests that something has been implemented here so
> it likely ought to work?

Yes, my gut feel is that qemu-img IS able to take advantage of
NBD_CMD_BLOCK_STATUS to avoid reading sparse portions of the source file
during qemu-img convert, so it IS worth figuring out how to get
structured replies and block status wired into nbdkit. It's on my list
of fun projects to attempt (but trailing behind other stuff like libvirt
incremental snapshots at the moment).

> 
> I guess the other question is if there are any limitations around
> alignment that we should be aware of.  It seems as if VDDK uses a
> larger cluster size for sparseness than its sector size (as does
> qcow2).

In general, block status will naturally report holes aligned to
granularity, but it really shouldn't matter to correctness (reporting
holes is merely an optimization, after all).

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <http://listman.redhat.com/archives/libguestfs/attachments/20190307/ca1f1918/attachment.sig>