<div dir="ltr">Just following up, this is indeed an apparmor issue. The attachment work with apparmor disabled. This is a libvirt bug: <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1361592">https://bugzilla.redhat.com/show_bug.cgi?id=1361592</a></div><div class="gmail_extra"> <div class="gmail_quote">On Mon, Aug 22, 2016 at 2:26 PM, Erlon Cruz <<a href="mailto:sombrafam@gmail.com" target="_blank">sombrafam@gmail.com</a>> wrote: <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">If that was the case I wouldn't be able to attach a volume without snapshots. But just in case I did the following test, before attaching the file, I copied all the file to a local folder and changed the path in XML with the new paths. The problem repeats: <div> </div><div>2016-08-22 17:14:19.795+0000: 3799: info : qemuMonitorIOProcess:423 : QEMU_MONITOR_IO_PROCESS: mon=0x7f68f800c4c0 buf={"return": "Could not open backing file: Could not open '/home/ubuntu/volumes/volume-d81f7e7e-4215-4433-aad2-50adcd31b29d.7a9ea169-a3ab-48bd-bd9d-e7585860abe4': Permission denied\r\n", "id": "libvirt-30"} </div><div> </div><div>But, if I use the top backing file with same permissions (/home/ubuntu/volumes/volume-d81f7e7e-4215-4433-aad2-50adcd31b29d), the operation suceeds. Can this be a libvirt/QEMU bug?</div><div> </div><div>Han, can you tell me more details about the commands you used and the versions of or libvirt/qemu? I couldn't reproduce from the steps you outlined.</div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"> <div class="gmail_quote">On Fri, Aug 19, 2016 at 12:02 AM, Han Han <<a href="mailto:hhan@redhat.com" target="_blank">hhan@redhat.com</a>> wrote: <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Erlon, Since you say you got Permission denied error while the ACL mode is 777， I guess it is due to security system like selinux or apparmor. As I know, selinux defaultly will disable qemu to use volume based on NFS. We need to use 'setsebool virt_use_nfs 1' to enable it. Well, your OS is ubuntu, so I think apparmor causes the permission problem. Try to find a way to enable virt_use_nfs or disable apparmor. Then check if the problem can be reproduced. Han ----- Original Message ----- From: "Erlon Cruz" <<a href="mailto:sombrafam@gmail.com" target="_blank">sombrafam@gmail.com</a>> To: "Han Han" <<a href="mailto:hhan@redhat.com" target="_blank">hhan@redhat.com</a>> Cc: "Yang Yang" <<a href="mailto:yanyang@redhat.com" target="_blank">yanyang@redhat.com</a>>, "Yuan Dan" <<a href="mailto:dyuan@redhat.com" target="_blank">dyuan@redhat.com</a>>, "Xuesong Zhang" <<a href="mailto:xuzhang@redhat.com" target="_blank">xuzhang@redhat.com</a>>, <a href="mailto:libvirt-users@redhat.com" target="_blank">libvirt-users@redhat.com</a> Sent: Thursday, 18 August, 2016 12:44:36 PM Subject: Re: [libvirt-users] Attaching disks with external snapshots Hi Han, Any update on this? Erlon On Wed, Aug 17, 2016 at 9:58 AM, Erlon Cruz <<a href="mailto:sombrafam@gmail.com" target="_blank">sombrafam@gmail.com</a>> wrote: > Han, > > Did the same test again with libvirt debug enabled. This time I added the > disk[1] without snaps (OK), and then added the disk[2] with snapshots. The > libvirt log[3] (line 320) says: > > *Could not open > '/opt/stack/data/nova/mnt/a3b4c6ddd9bf82edd4f726872be58d05/volume-a75b42e8-8175-4516-b724-43ee6f3d2520': > Permission denied\r\n", "id": "libvirt-33"}]* <div><div>> > But this file has 777 permissions. I can access it with any user I > tried[4]. Tough its mounted over NFS, the permissions seems to be right. > > Erlon > > [1] <a href="http://paste.openstack.org/show/556055/" rel="noreferrer" target="_blank">http://paste.openstack.org/show/556055/</a> > [2] <a href="http://paste.openstack.org/show/556056/" rel="noreferrer" target="_blank">http://paste.openstack.org/show/556056/</a> > [3] <a href="http://pastebin.com/CCFcKq5b" rel="noreferrer" target="_blank">http://pastebin.com/CCFcKq5b</a> > [4] <a href="http://paste.openstack.org/show/559018/" rel="noreferrer" target="_blank">http://paste.openstack.org/show/559018/</a> > > > On Tue, Aug 16, 2016 at 2:41 PM, Erlon Cruz <<a href="mailto:sombrafam@gmail.com" target="_blank">sombrafam@gmail.com</a>> wrote: > >> Hi Han, >> >> Thanks for helping. What is the XML before the snapshot is created? Can >> you post the XML files you used too? The files are in the links pointed by >> the numbers [1], [2] etc. >> >> The versions are: >> qemu-system-x86_64 --version >> QEMU emulator version 2.5.0 (Debian 1:2.5+dfsg-5ubuntu10.4), Copyright >> (c) 2003-2008 Fabrice Bellard >> >> libvirtd --version >> libvirtd (libvirt) 1.3.1 >> >> >> >> Erlon >> >> On Mon, Aug 15, 2016 at 12:56 AM, Han Han <<a href="mailto:hhan@redhat.com" target="_blank">hhan@redhat.com</a>> wrote: >> >>> Hi，Erlon >>> I was not able to reproduce your problem. It is ok to attach disk with >>> external snapshot. >>> I did it with following steps: >>> # qemu-img create /tmp/vdb 100M >>> Formatting '/tmp/vdb', fmt=raw size=104857600 >>> # virsh start snap >>> Domain snap started >>> # virsh snapshot-create-as snap s1 --disk-only --diskspec >>> vdb,file=/tmp/vdb.s1 --quiesce >>> Domain snapshot s1 created >>> # virsh dumpxml snap|awk '/<disk/,/<\/disk/' >>> <disk type='file' device='disk'> >>> <driver name='qemu' type='qcow2' cache='none'/> >>> <source file='/var/lib/libvirt/images/snap.s1'/> >>> <backingStore type='file' index='1'> >>> <format type='qcow2'/> >>> <source file='/var/lib/libvirt/images/snap.qcow2'/> >>> <backingStore/> >>> </backingStore> >>> <target dev='vda' bus='virtio'/> >>> <alias name='virtio-disk0'/> >>> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' >>> function='0x0'/> >>> </disk> >>> <disk type='file' device='disk'> >>> <driver name='qemu' type='qcow2'/> >>> <source file='/tmp/vdb.s1'/> >>> <backingStore type='file' index='1'> >>> <format type='raw'/> >>> <source file='/tmp/vdb'/> >>> <backingStore/> >>> </backingStore> >>> <target dev='vdb' bus='virtio'/> >>> <alias name='virtio-disk1'/> >>> <address type='pci' domain='0x0000' bus='0x00' slot='0x0c' >>> function='0x0'/> >>> </disk> >>> >>> Save the dumped vdb's xml to /tmp/vdb.xml >>> # virsh detach-device snap /tmp/vdb.xml >>> Device detached successfully >>> # virsh attach-device snap /tmp/vdb.xml >>> Device attached successfully >>> # virsh dumpxml snap|awk '/<disk/,/<\/disk/' >>> <disk type='file' device='disk'> >>> <driver name='qemu' type='qcow2' cache='none'/> >>> <source file='/var/lib/libvirt/images/snap.s1'/> >>> <backingStore type='file' index='1'> >>> <format type='qcow2'/> >>> <source file='/var/lib/libvirt/images/snap.qcow2'/> >>> <backingStore/> >>> </backingStore> >>> <target dev='vda' bus='virtio'/> >>> <alias name='virtio-disk0'/> >>> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' >>> function='0x0'/> >>> </disk> >>> <disk type='file' device='disk'> >>> <driver name='qemu' type='qcow2'/> >>> <source file='/tmp/vdb.s1'/> >>> <backingStore type='file' index='1'> >>> <format type='raw'/> >>> <source file='/tmp/vdb'/> >>> <backingStore/> >>> </backingStore> >>> <target dev='vdb' bus='virtio'/> >>> <alias name='virtio-disk1'/> >>> <address type='pci' domain='0x0000' bus='0x00' slot='0x0c' >>> function='0x0'/> >>> </disk> >>> >>> So I managed to attach disks with snapshot. >>> I want to know what is your libvirt and qemu-kvm version? >>> And can you provide your files: /tmp/disk.xml[1], >>> /tmp/snap-from-disk.xml[2], /tmp/disk-with-snap.xml[4], >>> /tmp/disk-with-snap.xml . I need more details. >>> >>> Han Han >>> >>> ----- Original Message ----- >>> From: "Erlon Cruz" <<a href="mailto:sombrafam@gmail.com" target="_blank">sombrafam@gmail.com</a>> >>> To: <a href="mailto:libvirt-users@redhat.com" target="_blank">libvirt-users@redhat.com</a> >>> Sent: Friday, August 12, 2016 10:23:24 AM >>> Subject: [libvirt-users] Attaching disks with external snapshots >>> >>> Hi folks, >>> >>> I'm trying to to attach a disk to an instance using libvirt. The problem >>> is, this disk has external snapshots. The process tried was: >>> >>> 1 - Attach a disk in the domain: >>> virsh# attach-device instance-00000006 /tmp/disk.xml[1] --live >>> >>> 2 - Snapshot the disk[2]: >>> virsh# snapshot-create instance-00000006 --quiesce --xmlfile >>> /tmp/snap-from-disk.xml[2] --disk-only >>> >>> 3 - Dump the domain XML and create a new disk file from it: >>> virsh# dumpxml instance-00000006 >>> ... [3] >>> >>> 4 - Dettach the device and re-add it using the new disk file: >>> virsh# detach-device instance-00000006 /tmp/disk-with-snap.xml[4] >>> Device detached successfully >>> virsh# attach-device instance-00000006 /tmp/disk-with-snap.xml >>> error: Failed to attach device from /tmp/disk-with-snap.xml >>> error: internal error: unable to execute QEMU command 'device_add': >>> Property 'virtio-blk-device.drive' can't find value 'drive-virtio-disk1' >>> >>> Question, is this operation supported? If yes, how is the correct >>> procedure? >>> >>> Erlon >>> >>> >>> [1] <a href="http://paste.openstack.org/show/556055/" rel="noreferrer" target="_blank">http://paste.openstack.org/show/556055/</a> >>> [2] <a href="http://paste.openstack.org/show/556056/" rel="noreferrer" target="_blank">http://paste.openstack.org/show/556056/</a> >>> [3] <a href="http://paste.openstack.org/show/556063/" rel="noreferrer" target="_blank">http://paste.openstack.org/show/556063/</a> >>> [4] <a href="http://paste.openstack.org/show/556064/" rel="noreferrer" target="_blank">http://paste.openstack.org/show/556064/</a> >>> >>> _______________________________________________ >>> libvirt-users mailing list >>> <a href="mailto:libvirt-users@redhat.com" target="_blank">libvirt-users@redhat.com</a> >>> <a href="https://www.redhat.com/mailman/listinfo/libvirt-users" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/libvirt-users</a> >>> >> >> > </div></div></blockquote></div> </div> </div></div></blockquote></div> </div>