[Libguestfs] [libvirt-users] virt-sparsify changing the apparent-size of files

Thu Jun 23 15:57:43 UTC 2016

Hello,

Thanks for all the information, I think my last remaining question is
why the image created with virt-install shows the full apparent size
and the image from virt-sparsify does not?  (The image testimage1.qcow2
created with default options).

As they are both qcow2 I would expect them both to have the same
behavior as far as apparent size goes.  Or is there an option I am
missing in virt-sparsify that would keep the full apparent size?  Or
perhaps one in virt-install to set apperent and real size the same?

On Thu, 23 Jun 2016 09:39:52 +0200
Andrea Bolognani <abologna at redhat.com> wrote:
> The '--format' option is to specify the image format for the
> input image. If you want the *output* image to be raw, you'll
> have to use '--convert raw'.

On Thu, 23 Jun 2016 08:52:35 +0100
"Richard W.M. Jones" <rjones at redhat.com> wrote:
> This is incorrect usage.  You've told virt-sparsify that the input
> image is raw.  But it's not raw, it's qcow2.

That makes sense, I won't try and tell it the input is raw anymore
when its not.

Keelan

On Thu, 23 Jun 2016 08:52:35 +0100
"Richard W.M. Jones" <rjones at redhat.com> wrote:

> On Wed, Jun 22, 2016 at 05:56:09PM -0500,
> libvirt_users at skagitattic.com wrote:
> > > Delete what you've done and start from the beginning.  Describe
> > > exactly how you created the guest.  Use 'qemu-img info' to show
> > > the format of the input file.  Show precisely the virt-sparsify
> > > command you are running.  And use 'qemu-img info' on the output
> > > file too.  
> > 
> > Okay, here goes.  Start off with an install 
> > 
> >     root at testingbox:~ # virt-install --name testimage --memory 512
> > -l
> > 'http://ftp.nl.debian.org/debian/dists/jessie/main/installer-amd64/'
> > --disk size=50 --nographics -x "console=ttyS0"             
> > 
> >     Starting install...
> >     Retrieving file
> >     MANIFEST...
> >     | 3.3 kB     00:00 ... Retrieving file
> >     linux...
> >     | 6.0 MB     00:03 ... Retrieving file
> >     initrd.gz...
> >     |  29 MB     00:09 ... Allocating
> >     'testimage.qcow2'
> >     |  50 GB     00:00 Creating
> >     domain...
> >     |    0 B     00:00 Connected to domain testimage Escape
> > character is [    0.000000] Initializing cgroup subsys cpuset
> > [    0.000000] Initializing cgroup subsys cpu [    0.000000]
> > Initializing cgroup subsys cpuacct [    0.000000] Linux version
> > 3.16.0-4-amd64 (debian-kernel at lists.debian.org) (gcc version 4.8.4
> > (Debian 4.8.4-1) ) #1 SMP Debian 3.16.7-ckt25-2 (2016-04-08)
> > 
> > ....   Many more lines here and it turns interactive.  Suffice it to
> > say I did a basic minimal install following the prompts.  Nothing
> > special here.
> > 
> > This created an image that is 2.5G with a 50G apparent size and
> > identifies as qcow2 as shown below.
> > 
> >     # ls -slh testimage.qcow2 
> >     2.5G -rw-r--r-- 1 root root 51G Jun 22 20:22 testimage.qcow2
> > 
> >     # qemu-img info testimage.qcow2
> >     image: testimage.qcow2
> >     file format: qcow2
> >     virtual size: 50G (53687091200 bytes)
> >     disk size: 2.4G
> >     cluster_size: 65536
> >     Format specific information:
> >         compat: 1.1
> >         lazy refcounts: true
> > 
> > Now we sparsify (/bigtmp is on a larger partition then / to give
> > virt-sparsify enough room to work with so it does not complain).
> > 
> >     root at testingbox: 08:59 PM # virt-sparsify testimage.qcow2
> >     testimage1.qcow2 --tmp /bigtmp Input disk virtual size =
> > 53687091200 bytes (50.0G) Create overlay file in /bigtmp to protect
> > source disk ... Examine source disk ...
> >        100%
> > ⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒⟧
> >     --:-- Fill free space in /dev/sda1 with zero ... 100%
> > ⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒⟧
> >     00:00 Clearing Linux swap on /dev/sda5 ... 100%
> > ⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒⟧
> >     --:-- Copy to destination and make sparse ...
> > 
> >     Sparsify operation completed with no errors.  Before deleting
> > the old disk, carefully check that the target disk boots and works
> >     correctly. 
> >     root at testingbox: 09:04 PM # 
> > 
> > This image reports still reports as qcow2 however the apparent and
> > real size is the same.
> > 
> >     # ls -slh testimage1.qcow2 
> >     1.1G -rw-r--r-- 1 root root 1.1G Jun 22 21:04 testimage1.qcow2
> > 
> >     # qemu-img info testimage1.qcow2
> >     image: testimage1.qcow2
> >     file format: qcow2
> >     virtual size: 50G (53687091200 bytes)
> >     disk size: 1.1G
> >     cluster_size: 65536
> >     Format specific information:
> >         compat: 1.1
> >         lazy refcounts: false  
> 
> virt-sparsify saved about 1.3 GB of disk space (2.4G -> 1.1G).
> 
> > If we try it again but specify raw its MUCH faster
> > 
> >     root at testingbox: 09:26 PM # virt-sparsify testimage.qcow2
> >     testimage2.qcow2 --tmp /bigtmp --format raw  
> 
> This is incorrect usage.  You've told virt-sparsify that the input
> image is raw.  But it's not raw, it's qcow2.
> 
> In any case, virt-sparsify opened the image as raw, couldn't make any
> sense of it (it appears to virt-sparsify to be random data, not a disk
> image), and so it cannot sparsify it properly.  This is "quicker" only
> because virt-sparsify didn't do any sparsification of filesystems,
> because it couldn't see the filesystems.
> 
> > This time it takes up more space and reports real and apparent size
> > differently.  It still reports as qcow2 with qemu-img.
> > 
> >     # ls -slh testimage2.qcow2 
> >     1.7G -rw-r--r-- 1 root root 51G Jun 22 21:27 testimage2.qcow2
> >     # qemu-img info testimage2.qcow2
> >     image: testimage2.qcow2
> >     file format: qcow2
> >     virtual size: 50G (53687091200 bytes)
> >     disk size: 1.7G  
> 
> Looks like virt-sparsify still found some clusters containing all
> zeroes which it could trim, even though it didn't know how to read the
> disk.
> 
> Anyway, the moral is, don't misinform virt-sparsify of the format
> of your input image.
> 
> If what you really intended to do was to convert the format from qcow2
> to raw, you should use '--convert raw' instead.
> 
> > The only things I see different from the finished files is that the
> > raw one uses a bit more space, the apparent size is not set to the
> > virtual size on the default sparsify one and the lazy refcounts is
> > different.
> > 
> > Is there something more detailed then "qemu-img info" that I could
> > tell the difference with?  
> 
> qemu-img info is the correct tool to use.
> 
> > > Raw has fewer features, especially no simple snapshotting or
> > > support for backing files.  
> 
> I wrote this thinking you were talking about actual raw format disks,
> not about qcow2 but where you've misinformed virt-sparsify about the
> format.
> 
> > Is simple snapshotting using qemu-img snapshot or something else?  I
> > tried making a snapshot with both test images and they both
> > appeared to work.  
> 
> http://wiki.qemu.org/Documentation/CreateSnapshot
> 
> >     # qemu-img snapshot -c test  testimage1.qcow2 
> >     # qemu-img snapshot -c test  testimage2.qcow2 
> >     # qemu-img info  testimage2.qcow2 
> >     image: testimage2.qcow2
> >     file format: qcow2
> >     virtual size: 50G (53687091200 bytes)
> >     disk size: 1.7G
> >     cluster_size: 65536
> >     Snapshot list:
> >     ID        TAG                 VM SIZE                DATE
> > VM CLOCK 1         test                      0 2016-06-22 22:41:52
> >      00:00:00.000 Format specific information:
> >         compat: 1.1
> >         lazy refcounts: true  
> 
> This works because testimage2.qcow2 is still in qcow2 format.
> 
> Anyway, use the correct --format option.
> 
> Rich.
> 