[libvirt-users] Researching why different cache modes result in 'some' guest filesystem corruption..

[vincent at cojot.name]

Hi All,

I've been chasing down an issue in recent weeks (my own lab, so no prod 
here) and I'm reaching out in case someone might have some guidance to 
share.

I'm running fairly large VMs (RHOSP underclouds - 8vcpu, 32gb ram, about 
200gb single disk as a growable qcow2) on some RHEL7.6 hypervisors (kernel 
3.10.0-927.2x.y, libvirt 4.5.0, qemu-kvm-1.5.3) on top of SSD/NVMe drives 
with various filesystems (vxfs, zfs, etc..) and using ECC RAM.

The issue can be described as follows:

- the guest VMs work fine for a while (days, weeks) but after a kernel
   update (z-stream) comes in, I am often greeted by the following message
   immediately after rebooting (or attempting to reboot into the new
   kernel):

"error: not a correct xfs inode"

- booting the previous kernel works fine and re-generating the initramfs
   for the new kernel (from the n-1 kernel) does not solve anything.

- if booted from an ISO, xfs_repair does not find errors.

- on ext4, there seems to be some kind of corruption there too.

I'm building the initial guest image qcow2 for those guest VMs this way:

1) start with a rhel-guest image (currently 
rhel-server-7.6-update-5-x86_64-kvm.qcow2)

2) convert to LVM by doing this:
  qemu-img create -f qcow2 -o preallocation=metadata,cluster_size=1048576,lazy_refcounts=off final_guest.qcow2 512G
  virt-format -a final_guest.qcow2 --partition=mbr --lvm=/dev/rootdg/lv_root --filesystem=xfs
  guestfish --ro -a rhel_guest.qcow2 -m /dev/sda1 -- tar-out / - | \
  guestfish --rw -a final_guest.qcow2 -m /dev/rootdg/lv_root  -- tar-in - /

3) use "final_guest.qcow2" as the basis for my guests with LVM.

After chasing down this issue some more and attempting various 
things (build the image on Fedora29, build a real XFS filesystem inside a 
VM and use the generated qcow2 as a basis instead of virt-format)..

..I've noticed that the SATA disk of each of those guests were using 
'directsync' (instead of 'Hypervisor Default'). As soon as I switched to 
'None', the XFS issues disappeared and I've now applied several 
consecutive kernel updates without issues. Also, 'directsync' or 
'writethrough', while providing decent performance, both exhibited the XFS 
'corruption' behaviour.. Only 'none' seem to have solved that.

I've read the docs but I thought it was OK to use those modes (UPS, 
Battery-Backed RAID, etc..)

Does anyone have any idea what's going on or what I may be doing wrong?

Thanks for reading,

Vincent