Libvirt + Debian Live = Heart Attack

[Elias Mobery]

Hey Martin, wow thanks for the detailed info!

I'll definitely look into this, but I kind of figured it out, I believe.

So, I keep the images in the read-only squashfs, boot the live system and
use snapshot writeback:
virsh snapshot-create-as with options --disk-only --diskspec

This pretty much fixes my problem, and only disk writes occur in tmpfs,
instead of the entire image being copied.

BUT, now I have to reduce the disk space within the  VM to almost 0 to save
space.

I'm reading up on qemu-img convert and virt-sparsify to accomplish this,
but I will also try your method now, it seems more professional.

Thank you for your amazing input and patience, I love this community.
Sharing this information is the literal definition of freedom IMO, thank
you all!

On Thu, Nov 25, 2021, 11:40 AM Martin Kletzander <mkletzan at redhat.com>
wrote:

> On Wed, Nov 24, 2021 at 06:21:22PM +0100, Elias Mobery wrote:
> >Hey Martin & Michal, thank you both for the replies!
> >
> >Yes, sorry I messed up that seclabel entry, thanks. I fixed it and there
> is
> >no more error, but the images are still copied unfortunately.
> >
> >@Michal yes, I have been reading the Debian Live Manual like a contract,
> >after messing with qemu.conf and permissions for days, I think its
> >something in the live build and not libvirt.
> >
> >@Martin yeah both virsh start and virt-manager cause the images to be
> >copied to /run..... first.
> >
> >Yes, when I boot the Debian Live ISO, plug in the USB (mounted rw), copy
> >the images to the live system from the USB and click run in virt-manager,
> >the VM starts instantly and no copies are made in /run....
> >
> >I use ISO loopback and toram to load the system directly into ram via
> tmpfs
> >mount. That's my grub.cfg
> >
> >menuentry "debian-live.iso" { set iso="/ISO/debian-live.iso" set
> >root=(hd0,5) loopback loop $iso linux (loop)/live/vmlinuz-4.19.0-6-amd64
> >boot=live components toram findiso=$iso initrd
> >(loop)/live/initrd.img-4.19.0-6-amd64 }
> >
> >So, the squashfs is mounted read-only in
> >/run/live/rootfs/filesystem.squashfs and its /dev/loop1
> >
> >/proc/mounts says:
> >
> >tmpfs /run/live/overlay rw
> >overlay rw  lowerdir=/run/live/rootfs/filesystem.squashfs
> >upperdir=/run/live/overlay/rw  (same place where the images are copied)
> >
>
> I do not think this is relevant, but sometimes overlayfs wants to have a
> workdir as well.
>
> >So you think that because the images are part of the squashfs which is
> >read-only, they are first copied to /run/live/overlay/rw... to be written
> >to?
> >
>
> Are they really copied?  I can't think of many more things why this
> would happen.  Maybe some locking code does that in the backend (both
> libvirt and qemu lock parts of the disks).
>
> >I always thought that tmpfs is mounted on top of squash, so it's as if the
> >squashfs were writable, without having to literally copy between the two
> >filesystems.
> >
>
> It is not mounted over, if you mount over something you lose the
> lowerdir, that's why it needs to be done with overlayfs which does the
> copy-on-write.
>
> That leads me to another idea.  Since you want copy-on-write for the
> images, I presume, the image will get copied anyway once the guest
> writes even a single byte to the image.  Unfortunately overlayfs is not
> the best way for this scenario due to the granularity it offers
> (per-file, not per-block or anything smaller).  What I think you
> actually want in this case is to have a qcow image with the original
> image used as backing.  You can create those with libvirt or even
> without it, just using qemu-img, for example like this:
>
>    qemu-img create -b /path/to/non-modified.image -f qcow2 -F
>    format_of_backing_file /path/to/new-image.qcow2
>
> For more information please read the qemu-img manual page to make sure
> you use it properly, I just wrote the above from the top of my head, not
> tested it.  The backing file path can also be relative.
>
> If you specify the new image for the machines then the granularity
> changes to way smaller blocks than the whole file making it occupy
> less space in case of changes.
>
> >Also I made the same build with Virtualbox and there is no copying of the
> >images to /run/live/overlay.
> >
> >I will do some serious research into squashfs and tmpfs rn.
> >
>
> If you want to figure out when it happens I would suggest enabling debug
> logs for libvirt, starting the qemu VM with --paused, but maybe it will
> need some more debugging on lower layers.  Anyway I really believe the
> approach suggested above is what will lead to better results for you.
>
> Have a nice day
> Martin
>
> >
> >
> >
> >On Wed, Nov 24, 2021, 5:10 PM Martin Kletzander <mkletzan at redhat.com>
> wrote:
> >
> >> On Wed, Nov 24, 2021 at 04:01:34PM +0100, Elias Mobery wrote:
> >> >Hello Michal, thank you for the reply!
> >> >
> >> >I've carefully tested everything you suggested, thanks.
> >> >
> >> >I set dynamic_ownership=0 and use these hooks during the live build for
> >> >permissions. (I googled a lot, and apparently libvirt needs the images
> to
> >> >be executable too)
> >> >
> >> >chown -R libvirt-qemu:kvm  /var/lib/libvirt/images
> >> >chmod -R g+rwx /var/lib/libvirt/images
> >> >
> >>
> >> I do not see why would the images need to be executable, you might need
> >> an executable bit set on the directory so that your and qemu user can
> >> browse it.
> >>
> >> >Booting the live debian iso everything works in virt-manager, but
> again,
> >> >after clicking "run", a copy of the vm image is created in
> >> >/run/live/overlay/rw/var/lib/libvirt/images and only then does the VM
> >> start.
> >> >
> >>
> >> Does that happen when you try to run it with virsh instead of
> >> virt-manager?  Are you connected to the system daemon?
> >>
> >> >Either it's still being chowned or chmodded somehow, or it's something
> >> >else, but I can't stop this copy being made.
> >> >
> >> >Interestingly, when I boot the Live debian iso and then copy the images
> >> >into /var/lib/libvirt/images from my USB stick, the VM starts
> immediately
> >> >without creating any copies in the /run/live.... directory. So my
> guess is
> >> >that maybe the squashfs could be the issue?
> >> >
> >>
> >> Oh, interesting, is the USB stick filesystem mounted R/W and the live
> >> ISO filesystem mounted read-only?  How is the overlay mounted?
> >>
> >> >Editing the XML
> >> >
> >> ><source file='/var/lib/libvirt/images/vm1.qcow2'>
> >> >      <seclabel relabel='no'/>
> >> >    </source>
> >> >
> >> >This results in an error:
> >> >Unsupported Configuration:
> >> >Security driver model 'null' not available
> >> >
> >>
> >> For issues like this it is most helpful to check the documentation:
> >>
> >>    https://libvirt.org/formatdomain.html
> >>
> >> where you can see it is just a missing attribute `model`, in this case
> >> model="dac".
> >>
> >> >Here I tried setting security_driver=none in qemu.conf but same error
> >> after.
> >> >
> >> ></devices>
> >> >    <seclabel type='none'/>
> >> >  </domain>
> >> >
> >>
> >> Same here.
> >>
> >> >This also returns an Error but I'm still googling to understand it
> >> properly.
> >> >
> >> >XML document failed to validate against schema
> >> >Unable to validate doc against /usr/share/libvirt/schemas/domain.rng
> >> >Invalid element relabel  for element seclabel
> >> >Extra element seclabel in interleave
> >> >Element domain failed to validate content
> >> >
> >> >Thanks again so much for your helo, I've been messing with this for
> weeks
> >> >now and it's killing me.
> >> >
> >> >On Tue, Nov 23, 2021, 9:43 PM Michal Prívozník <mprivozn at redhat.com>
> >> wrote:
> >> >
> >> >> On 11/23/21 17:25, Elias Mobery wrote:
> >> >> >
> >> >> > Hi everyone!
> >> >> >
> >> >> > I've built a Debian Live ISO with packages qemu and libvirt to run
> a
> >> VM
> >> >> > in the live environment.
> >> >> >
> >> >> > The guest images are placed in  /var/lib/libvirt/images and 2GB
> each.
> >> >> >
> >> >> > Everything works great, except for one issue.
> >> >> >
> >> >> > When starting a VM, libvirt automatically issues a chown command to
> >> the
> >> >> > images, changing ownership.
> >> >> >
> >> >> > This results in a copy of the images being created in
> >> >> > /run/live/overlay/rw/var/lib/libvirt/images
> >> >> >
> >> >> > I don't want these copies to be made but can't stop it.
> >> >> >
> >> >> > I've tried editing qemu.conf user/group, dynamic ownership etc.
> >> without
> >> >> > any luck.
> >> >> >
> >> >> > Is there a way to STOP libvirt from changing the ownership of these
> >> >> images?
> >> >> >
> >> >> >
> >> >>
> >> >> Setting dynamic_ownership=0 in qemu.conf is the way to go (don't
> forget
> >> >> to restart the daemon after you made the change).
> >> >>
> >> >> Alternatively, you can set <seclabel/> either for whole domain or
> >> >> individual disks, e.g. like this:
> >> >>
> >> >>   <disk type='file' device='disk'>
> >> >>     <driver name='qemu' type='qcow2'/>
> >> >>     <source file='/var/lib/libvirt/images/vm1.qcow2'>
> >> >>       <seclabel relabel='no'/>
> >> >>     </source>
> >> >>   </disk>
> >> >>
> >> >> or for whole domain:
> >> >>
> >> >>     ...
> >> >>     </devices>
> >> >>     <seclabel type='none'/>
> >> >>   </domain>
> >> >>
> >> >> I'm not sure what your setup is, but if chown() is a problem then
> what
> >> >> if guest tries to write onto its disk? Wouldn't that create a copy in
> >> >> overlay?
> >> >>
> >> >> Michal
> >> >>
> >> >>
> >>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/libvirt-users/attachments/20211125/611ac588/attachment.htm>