[Libguestfs] [PATCH] v2v: Add --print-estimate option to print source size estimate.

[Richard W.M. Jones]

On Wed, Aug 15, 2018 at 02:53:05PM +0200, Kevin Wolf wrote:
> [ Cc: qemu-block ]
> 
> Am 15.08.2018 um 12:55 hat Richard W.M. Jones geschrieben:
> > (Adding Stefan who implemented the subcommand)
> > 
> > On Wed, Aug 15, 2018 at 12:44:44PM +0200, Kevin Wolf wrote:
> > > Am 15.08.2018 um 12:26 hat Richard W.M. Jones geschrieben:
> > > > On Tue, Aug 14, 2018 at 09:31:06PM +0300, Nir Soffer wrote:
> > > > > On Tue, Aug 14, 2018 at 8:29 PM Richard W.M. Jones <rjones at redhat.com>
> > > > > wrote:
> > > > > 
> > > > > > This option prints the estimated size of the data that will be copied
> > > > > > from the source disk.
> > > > > >
> > > > > > For interest, the test prints:
> > > > > >
> > > > > > 3747840 ../test-data/phony-guests/windows.img
> > > > > > Estimate: 3710976
> > > > > >
> > > > > 
> > > > > Why not use qemu-img measure on the overlay?
> > > > 
> > > > Yes this would be better, but oddly it doesn't work properly when
> > > > the output format is set to 'raw':
> > > > 
> > > >   /usr/bin/qemu-img 'measure' '-f' 'qcow2' '-O' 'raw' '--output=human' '/home/rjones/d/libguestfs/tmp/v2vovl62b7c2.qcow2'
> > > >   required size: 6442450944
> > > >   fully allocated size: 6442450944
> > > > 
> > > > However it's OK if the output format is set to 'qcow2':
> > > > 
> > > >   /usr/bin/qemu-img 'measure' '-f' 'qcow2' '-O' 'qcow2' '--output=human' '/home/rjones/d/libguestfs/tmp/v2vovla53d7c.qcow2'
> > > >   required size: 1047986176
> > > >   fully allocated size: 6443696128
> > > > 
> > > > I guess it ignores sparseness of raw images, but I can't see a way to
> > > > specify that on the command line.
> > > > 
> > > > OTOH the qcow2 figure is probably a good enough guess for our purposes
> > > > (ie. estimating how much data will be copied).
> > > 
> > > 'qemu-img measure' calculates the resulting file size, not the number of
> > > used blocks. I think this is because its main purpose is to create block
> > > devices (like LVs) of the right size, so sparseness on a filesystem
> > > level doesn't buy you anything.
> > 
> > But if I run ‘qemu-img convert ... -O raw output.raw’ then output.raw
> > will be a sparse file, and that's the file size I'd expect measure to
> > give us for "required size" (of course "fully allocated size" would be
> > the virtual size, and that is correct).
> > 
> > It does look as if the block/raw-format.c:raw_measure function is wrong.
> 
> No, it is correct. Its output is what the file size will be. For raw
> images, that is the same as the virtual disk size.
> 
> Not all blocks in the file will be actually allocated, but the file size
> is what 'ls -l' prints, not what 'du' prints (for regular files).
> 
> It becomes even clearer when you create LVs as the target. If you have a
> 10 GB image in which only the last 1 MB actually contains data, you
> still need a 10 GB volume. You can't create a 1 MB volume and then store
> data at an offset 10 GB - 1 MB, that would be way after the end of the
> block device.
> 
> > In any case we can use the qcow2 estimate for our purposes as it will
> > be near enough what we need (a rough estimate of the size of the copy).
> 
> I don't know what the exact purpose is, and it feels a bit hacky, but it
> might be good enough.

The original goal is to try to get an estimate for how many bytes a
subsequent ‘qemu-img convert’ will actually copy.  The estimate does
not need to be very precise: it is used so that we can make a
prediction of how long the copy will take.  The prediction is used for
planning purposes, such as whether the copy will fit into a planned
downtime window and how long the overall process (which involves
multiple hundreds of qemu-img convert copies) will take.

I think the qcow2 number is near enough for these purposes.

> I suppose what you really want is that 'qemu-img measure' provides
> another number for the space taken by allocated blocks. (Probably
> excluding metadata for non-raw formats? Might depend on the actual
> purpose.)
> 
> Kevin

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v