[Libguestfs] [PATCH libnbd] PROPOSAL Add nbdcp (NBD copying) tool.

Thu Jan 23 10:40:25 UTC 2020

On Wed, Jan 22, 2020 at 02:40:37PM +0000, Richard W.M. Jones wrote:
>This is a proposal for an NBD to/from file copying tool (not actually
>written).  Obviously this would duplicate functionality which is
>already available in qemu-img convert.
>
>The reasons for writing this tool would be:
>
> - to produce a tool which is very focused on the specific needs of
>   virt-v2v and similar migration scenarios
>
> - to have a small tool with minimal dependencies
>
> - fix some of the obvious problems with qemu-img convert like how it
>   handles devices which are already zeroed, and preallocation
>
> - enable optimizations which make sense for NBD but which might not
>   be applicable to qemu (eg. multi-conn, freely writing out of order
>   from multiple threads).
>
>Alternative is of course to not write a new tool and instead try to
>fix all this stuff in qemu-img itself.
>
>Anyway let me know your thoughts.
>

I like the proposal, there is only one thing I do not completely understand,
whether `-a sparse -S 0` means to just preserver the sparseness based on
whatever the server provides.

It would also be nice to have an option to specify a list of blocks to copy from
a file for example.  Although it might be a niche feature and people should just
use libnbd at that point.

>Rich.
>
>
>nbdcp(1)                            LIBNBD                            nbdcp(1)
>
>NAME
>       nbdcp - copy between NBD servers and local files
>
>SYNOPSIS
>        nbdcp [-a|--target-allocation allocated|sparse]
>              [-m|--multi-conn <n>] [-M|--multi-conn-target <n>]
>              [-p|--progress-bar] [-S|--sparse-detect <n>]
>              [-T|--threads <n>] [-z|--target-is-zero]
>              'nbd://...'|DISK.IMG 'nbd://...'|DISK.IMG
>
>DESCRIPTION
>       nbdcp is a utility that can copy quickly between NBD servers and local
>       raw format files (or block devices).  It can copy:
>
>       from NBD server to file (or block device)
>           For example, this command copies from the NBD server listening on
>           port 10809 on "example.com" to a local file called disk.img:
>
>            nbdcp nbd://example.com disk.img
>
>       from file (or block device) to NBD server
>           For example, this command copies from a local block device /dev/sda
>           to the NBD server listening on Unix domain socket /tmp/socket:
>
>            nbdcp /dev/sda 'nbd+unix:///?socket=/tmp/socket'
>
>       from NBD server to NBD server
>           For example this copies between two different exports on the same
>           NBD server:
>
>            nbdcp nbd://example.com/export1 nbd://example.com/export2
>
>       This program cannot: copy from file to file (use cp(1) or dd(1)), copy
>       to or from formats other than raw (use qemu-img(1) convert), or access
>       servers other than NBD servers (also use qemu-img(1)).
>
>       NBD servers are specified by their URI, following the NBD URI standard
>       at https://github.com/NetworkBlockDevice/nbd/blob/master/doc/uri.md
>
>   Controlling sparseness or preallocation in the target
>       The options -a (--target-allocation), -S (--sparse-detect) and -z
>       (--target-is-zero) together control sparseness in the target file.
>
>       By default nbdcp tries to both preserve sparseness from the source and
>       will detect runs of allocated zeroes and turn them into sparseness.  To
>       turn off detection of sparseness use "-S 0".
>
>       The -z option should be used if and only if you know that the target
>       block device is zeroed already.  This allows an important optimization
>       where nbdcp can skip zeroing or trimming parts of the disk that are
>       already zero.
>
>       The -a option is used to control the desired final preallocation state
>       of the target.  The default is "-a sparse" which makes the target as
>       sparse as possible.  "-a allocated" makes the target fully allocated.
>
>OPTIONS
>       --help
>           Display brief command line help and exit.
>
>       -a allocated
>       --target-allocation=allocated
>           Make the target fully allocated.
>
>       -a sparse
>       --target-allocation=sparse
>           Make the target as sparse as possible.  This is the default.  See
>           also "Controlling sparseness or preallocation in the target".
>
>       -m N
>       --multi-conn=N
>           Enable NBD multi-conn with up to "N" connections.  Only some NBD
>           servers support this but it can greatly improve performance.
>
>           The default is to enable multi-conn if we detect that the server
>           supports it, with up to 4 connections.
>
>       -M N
>       --multi-conn-target=N
>           If you are copying between NBD servers, use -m to control the
>           multi-conn setting for the source server, and this option (-M) to
>           control the multi-conn setting for the target server.
>
>       -p
>       --progress-bar
>           Display a progress bar during copying.
>
>       -p machine:FD
>       --progress-bar=machine:FD
>           Write a machine-readable progress bar to file descriptor "FD".
>           This progress bar prints lines with the format "COPIED/TOTAL"
>           (where "COPIED" and "TOTAL" are 64 bit unsigned integers).
>
>       -S 0
>       --sparse-detect=0
>           Turn off sparseness detection.
>
>       -S N
>       --sparse-detect=N
>           Detect runs of zero bytes of at least size "N" bytes and turn them
>           into sparse blocks on the target (if "-a sparse" is used).  This is
>           the default, with a 512 byte block size.
>
>       -T N
>       --threads N
>           Use at most "N" threads when copying.  Usually more threads leads
>           to better performance, up to the limit of the number of cores on
>           your machine and the parallelism of the underlying disk or network.
>           The default is to use the number of online processors.
>
>       -z
>       --target-is-zero
>           Declare that the target block device contains only zero bytes (or
>           sparseness that reads back as zeroes).  You must only use this
>           option if you are sure that this is true, since it means that nbdcp
>           will enable an optimization where it skips zeroing parts of the
>           disk that are zero on the source.
>
>       -V
>       --version
>           Display the package name and version and exit.
>
>SEE ALSO
>       qemu-img(1), libnbd(3), nbdsh(1).
>
>AUTHORS
>       Richard W.M. Jones
>
>COPYRIGHT
>       Copyright (C) 2020 Red Hat Inc.
>
>LICENSE
>       This library is free software; you can redistribute it and/or modify it
>       under the terms of the GNU Lesser General Public License as published
>       by the Free Software Foundation; either version 2 of the License, or
>       (at your option) any later version.
>
>       This library is distributed in the hope that it will be useful, but
>       WITHOUT ANY WARRANTY; without even the implied warranty of
>       MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
>       Lesser General Public License for more details.
>
>       You should have received a copy of the GNU Lesser General Public
>       License along with this library; if not, write to the Free Software
>       Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
>       02110-1301 USA
>
>libnbd-1.3.1                      2020-01-22                          nbdcp(1)
>
>
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/libguestfs/attachments/20200123/998925fe/attachment.sig>