<div dir="auto"><div>Hey Martin, wow thanks for the detailed info!<div dir="auto"> </div><div dir="auto">I'll definitely look into this, but I kind of figured it out, I believe.</div><div dir="auto"> </div><div dir="auto">So, I keep the images in the read-only squashfs, boot the live system and use snapshot writeback:</div><div dir="auto">virsh snapshot-create-as with options --disk-only --diskspec</div><div dir="auto"> </div><div dir="auto">This pretty much fixes my problem, and only disk writes occur in tmpfs, instead of the entire image being copied.</div><div dir="auto"> </div><div dir="auto">BUT, now I have to reduce the disk space within the VM to almost 0 to save space. </div><div dir="auto"> </div><div dir="auto">I'm reading up on qemu-img convert and virt-sparsify to accomplish this, but I will also try your method now, it seems more professional.</div><div dir="auto"> </div><div dir="auto">Thank you for your amazing input and patience, I love this community. Sharing this information is the literal definition of freedom IMO, thank you all!</div> <div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Nov 25, 2021, 11:40 AM Martin Kletzander <<a href="mailto:mkletzan@redhat.com" target="_blank" rel="noreferrer">mkletzan@redhat.com</a>> wrote: </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On Wed, Nov 24, 2021 at 06:21:22PM +0100, Elias Mobery wrote: >Hey Martin & Michal, thank you both for the replies! > >Yes, sorry I messed up that seclabel entry, thanks. I fixed it and there is >no more error, but the images are still copied unfortunately. > >@Michal yes, I have been reading the Debian Live Manual like a contract, >after messing with qemu.conf and permissions for days, I think its >something in the live build and not libvirt. > >@Martin yeah both virsh start and virt-manager cause the images to be >copied to /run..... first. > >Yes, when I boot the Debian Live ISO, plug in the USB (mounted rw), copy >the images to the live system from the USB and click run in virt-manager, >the VM starts instantly and no copies are made in /run.... > >I use ISO loopback and toram to load the system directly into ram via tmpfs >mount. That's my grub.cfg > >menuentry "debian-live.iso" { set iso="/ISO/debian-live.iso" set >root=(hd0,5) loopback loop $iso linux (loop)/live/vmlinuz-4.19.0-6-amd64 >boot=live components toram findiso=$iso initrd >(loop)/live/initrd.img-4.19.0-6-amd64 } > >So, the squashfs is mounted read-only in >/run/live/rootfs/filesystem.squashfs and its /dev/loop1 > >/proc/mounts says: > >tmpfs /run/live/overlay rw >overlay rw lowerdir=/run/live/rootfs/filesystem.squashfs >upperdir=/run/live/overlay/rw (same place where the images are copied) > I do not think this is relevant, but sometimes overlayfs wants to have a workdir as well. >So you think that because the images are part of the squashfs which is >read-only, they are first copied to /run/live/overlay/rw... to be written >to? > Are they really copied? I can't think of many more things why this would happen. Maybe some locking code does that in the backend (both libvirt and qemu lock parts of the disks). >I always thought that tmpfs is mounted on top of squash, so it's as if the >squashfs were writable, without having to literally copy between the two >filesystems. > It is not mounted over, if you mount over something you lose the lowerdir, that's why it needs to be done with overlayfs which does the copy-on-write. That leads me to another idea. Since you want copy-on-write for the images, I presume, the image will get copied anyway once the guest writes even a single byte to the image. Unfortunately overlayfs is not the best way for this scenario due to the granularity it offers (per-file, not per-block or anything smaller). What I think you actually want in this case is to have a qcow image with the original image used as backing. You can create those with libvirt or even without it, just using qemu-img, for example like this: qemu-img create -b /path/to/non-modified.image -f qcow2 -F format_of_backing_file /path/to/new-image.qcow2 For more information please read the qemu-img manual page to make sure you use it properly, I just wrote the above from the top of my head, not tested it. The backing file path can also be relative. If you specify the new image for the machines then the granularity changes to way smaller blocks than the whole file making it occupy less space in case of changes. >Also I made the same build with Virtualbox and there is no copying of the >images to /run/live/overlay. > >I will do some serious research into squashfs and tmpfs rn. > If you want to figure out when it happens I would suggest enabling debug logs for libvirt, starting the qemu VM with --paused, but maybe it will need some more debugging on lower layers. Anyway I really believe the approach suggested above is what will lead to better results for you. Have a nice day Martin > > > >On Wed, Nov 24, 2021, 5:10 PM Martin Kletzander <<a href="mailto:mkletzan@redhat.com" rel="noreferrer noreferrer" target="_blank">mkletzan@redhat.com</a>> wrote: > >> On Wed, Nov 24, 2021 at 04:01:34PM +0100, Elias Mobery wrote: >> >Hello Michal, thank you for the reply! >> > >> >I've carefully tested everything you suggested, thanks. >> > >> >I set dynamic_ownership=0 and use these hooks during the live build for >> >permissions. (I googled a lot, and apparently libvirt needs the images to >> >be executable too) >> > >> >chown -R libvirt-qemu:kvm /var/lib/libvirt/images >> >chmod -R g+rwx /var/lib/libvirt/images >> > >> >> I do not see why would the images need to be executable, you might need >> an executable bit set on the directory so that your and qemu user can >> browse it. >> >> >Booting the live debian iso everything works in virt-manager, but again, >> >after clicking "run", a copy of the vm image is created in >> >/run/live/overlay/rw/var/lib/libvirt/images and only then does the VM >> start. >> > >> >> Does that happen when you try to run it with virsh instead of >> virt-manager? Are you connected to the system daemon? >> >> >Either it's still being chowned or chmodded somehow, or it's something >> >else, but I can't stop this copy being made. >> > >> >Interestingly, when I boot the Live debian iso and then copy the images >> >into /var/lib/libvirt/images from my USB stick, the VM starts immediately >> >without creating any copies in the /run/live.... directory. So my guess is >> >that maybe the squashfs could be the issue? >> > >> >> Oh, interesting, is the USB stick filesystem mounted R/W and the live >> ISO filesystem mounted read-only? How is the overlay mounted? >> >> >Editing the XML >> > >> ><source file='/var/lib/libvirt/images/vm1.qcow2'> >> > <seclabel relabel='no'/> >> > </source> >> > >> >This results in an error: >> >Unsupported Configuration: >> >Security driver model 'null' not available >> > >> >> For issues like this it is most helpful to check the documentation: >> >> <a href="https://libvirt.org/formatdomain.html" rel="noreferrer noreferrer noreferrer" target="_blank">https://libvirt.org/formatdomain.html</a> >> >> where you can see it is just a missing attribute `model`, in this case >> model="dac". >> >> >Here I tried setting security_driver=none in qemu.conf but same error >> after. >> > >> ></devices> >> > <seclabel type='none'/> >> > </domain> >> > >> >> Same here. >> >> >This also returns an Error but I'm still googling to understand it >> properly. >> > >> >XML document failed to validate against schema >> >Unable to validate doc against /usr/share/libvirt/schemas/domain.rng >> >Invalid element relabel for element seclabel >> >Extra element seclabel in interleave >> >Element domain failed to validate content >> > >> >Thanks again so much for your helo, I've been messing with this for weeks >> >now and it's killing me. >> > >> >On Tue, Nov 23, 2021, 9:43 PM Michal Prívozník <<a href="mailto:mprivozn@redhat.com" rel="noreferrer noreferrer" target="_blank">mprivozn@redhat.com</a>> >> wrote: >> > >> >> On 11/23/21 17:25, Elias Mobery wrote: >> >> > >> >> > Hi everyone! >> >> > >> >> > I've built a Debian Live ISO with packages qemu and libvirt to run a >> VM >> >> > in the live environment. >> >> > >> >> > The guest images are placed in /var/lib/libvirt/images and 2GB each. >> >> > >> >> > Everything works great, except for one issue. >> >> > >> >> > When starting a VM, libvirt automatically issues a chown command to >> the >> >> > images, changing ownership. >> >> > >> >> > This results in a copy of the images being created in >> >> > /run/live/overlay/rw/var/lib/libvirt/images >> >> > >> >> > I don't want these copies to be made but can't stop it. >> >> > >> >> > I've tried editing qemu.conf user/group, dynamic ownership etc. >> without >> >> > any luck. >> >> > >> >> > Is there a way to STOP libvirt from changing the ownership of these >> >> images? >> >> > >> >> > >> >> >> >> Setting dynamic_ownership=0 in qemu.conf is the way to go (don't forget >> >> to restart the daemon after you made the change). >> >> >> >> Alternatively, you can set <seclabel/> either for whole domain or >> >> individual disks, e.g. like this: >> >> >> >> <disk type='file' device='disk'> >> >> <driver name='qemu' type='qcow2'/> >> >> <source file='/var/lib/libvirt/images/vm1.qcow2'> >> >> <seclabel relabel='no'/> >> >> </source> >> >> </disk> >> >> >> >> or for whole domain: >> >> >> >> ... >> >> </devices> >> >> <seclabel type='none'/> >> >> </domain> >> >> >> >> I'm not sure what your setup is, but if chown() is a problem then what >> >> if guest tries to write onto its disk? Wouldn't that create a copy in >> >> overlay? >> >> >> >> Michal >> >> >> >> >> </blockquote></div></div></div>