[libvirt-users] virtlock - a VM goes read-only

[Branimir Pejakovic]

Dear colleagues,

I am facing a problem that has been troubling me for last week and a half.
Please if you are able to help or offer some guidance.

I have a non-prod POC environment with 2 CentOS7 fully updated hypervisors
and an NFS filer that serves as a VM image storage. The overall environment
works exceptionally well. However, starting a few weeks ago I have been
trying to implement virtlock in order to prevent a VM running on 2
hypervisors at the same time.

Here is the description how the environment looks like in terms of virtlock
configuration on both hypervisors:

-- Content of /etc/libvirt/qemu.conf --
lock_manager = "lockd"

Only the above line is uncommented for direct locking.

# libvirtd --version; python -c "import platform;
print(platform.platform())"; virtlockd -V
libvirtd (libvirt) 3.2.0
Linux-3.10.0-693.2.2.el7.x86_64-x86_64-with-centos-7.4.1708-Core
virtlockd (libvirt) 3.2.0

# getenforce
Permissive

Here is the issue:

h1 # virsh list
 Id    Name                           State
----------------------------------------------------
 1     test09                         running

h1 # virsh domblklist test09
Target     Source
------------------------------------------------
vda        /storage_nfs/images_001/test09.qcow2


h1 #

h2 # virsh list
 Id    Name                           State
----------------------------------------------------

h2 # virsh list --all | grep test09
 -     test09                         shut off

h2 # virsh start test09
error: Failed to start domain test09
error: resource busy: Lockspace resource
'/storage_nfs/images_001/test09.qcow2' is locked

h2 # virsh list
 Id    Name                           State
----------------------------------------------------

h2 #

Before I start test09 I open a console to the guest and observe what is
going on in it. Once I try to start test09 (and get a message about locked
resource) on h2 hypervisor, I can see the following messages in the console
and the vm goes to ro mode:

on test09's console:

[  567.394148] blk_update_request: I/O error, dev vda, sector 13296056
[  567.395883] blk_update_request: I/O error, dev vda, sector 13296056

[  572.871905] blk_update_request: I/O error, dev vda, sector 8654040
[  572.872627] Aborting journal on device vda1-8.
[  572.873978] blk_update_request: I/O error, dev vda, sector 8652800
[  572.874707] Buffer I/O error on dev vda1, logical block 1081344, lost
sync page write
[  572.875472] blk_update_request: I/O error, dev vda, sector 2048
[  572.876009] Buffer I/O error on dev vda1, logical block 0, lost sync
page write
[  572.876727] EXT4-fs error (device vda1): ext4_journal_check_start:56:
Detected aborted journal[  572.878061] JBD2: Error -5 detected when
updating journal superblock for vda1-8.

[  572.878807] EXT4-fs (vda1): Remounting filesystem read-only
[  572.879311] EXT4-fs (vda1): previous I/O error to superblock detected
[  572.880937] blk_update_request: I/O error, dev vda, sector 2048
[  572.881538] Buffer I/O error on dev vda1, logical block 0, lost sync
page write

I also observe the guests'log:

-- /var/log/libvirt/qemu/test09.log --

block I/O error in device 'drive-virtio-disk0': Permission denied (13)
block I/O error in device 'drive-virtio-disk0': Permission denied (13)
block I/O error in device 'drive-virtio-disk0': Permission denied (13)
block I/O error in device 'drive-virtio-disk0': Permission denied (13)
block I/O error in device 'drive-virtio-disk0': Permission denied (13)
block I/O error in device 'drive-virtio-disk0': Permission denied (13)
block I/O error in device 'drive-virtio-disk0': Permission denied (13)

If it helps, here is the disk portion of an XML file:

    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/storage_nfs/images_001/test09.qcow2'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
function='0x0'/>
    </disk>

I usually do implement SELinux on a hypervisor to isolate guests even
further but this time I set it to permissive mode just to rule out SELinux
factor. The same thing happens when SELinux is in enforcing mode
(virt_use_nfs is set to on in that case) and audit2why doesn't report any
anomalies when parsing audit logs.

I have tried to use indirect locking via the same filer and with a
separated export for the hashes by removing the comment in
/etc/libvirt/qemu-lockd.conf for the following line:

file_lockspace_dir = "/var/lib/libvirt/lockd/files"

In this case the hashes are normally created on the NFS export mounted
under /var/lib/libvirt/lockd/files. I have also tried playing with both
QCOW2 and raw disk images for VMs (and even with XFS/ext4 based guests) but
the outcome is always the same. I have a couple of KVM books - consulted
them on this topic, consulted  Red Hat and SUSE docs but pretty much the
configuration instructions are, naturally, the same. I saw that  some
colleagues posted a few emails (ie
https://www.redhat.com/archives/libvirt-users/2015-September/msg00004.html)
to the list related to virtlock but it seems that it is not the same issue.
I have also, as a last resort, completely disabled SELinux, rebooted both
hypervisors, created a new vm, repeated all the steps listed above but with
the same results.

Now, I am pretty sure that I am missing something simple here since this is
a standard feature and should work out of the box if set correctly but so
far I cannot see what I am missing.

I would really appreciate any tip/help.

Thank you very much!!


Regards,

Branimir
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/libvirt-users/attachments/20171115/14b9cb93/attachment.htm>