[libvirt-users] Secondary disk unattached if guest unresponsive and killed with 'virsh destroy'

[Mathew Crane]

I have a very strange issue that has plagued our KVM-based hypervisor
infrastructure.

Environment:
- All EL7-based libvirt releases since EL7.0
- Fully patched EL7-based KVM guest on a fully patched EL7-based KVM host.
- Secondary disk attached via 'virsh attach-device guest
/vm/guest-with-secondary-disk-01/disk2.xml --persistent' (note --persistent
flag here)

Problem Description:
- Guest becomes unresponsive/hangs due to issue on guest. Typically the
scenario is that PID1 hangs or goes into a bad state due to a misbehaving
application.
- Before virsh destroy, 'virsh dumpxml guest | grep vd[ab]' returns the
expected filesystems.
- Guest is destroyed from the hypervisor via 'virsh destroy guest'.
- With guest still powered off, 'virsh dumpxml guest | grep vd[ab]' only
shows 'vda'. 'vdb' now unattached.
- To fix, requires re-run of 'virsh attach-device guest
guest-with-secondary-disk-01/disk2.xml --persistent'

Expected Behavior:
- 'virsh destroy guest' does not drop the secondary disk attached to the
guest if the node is in a hung state.

Other notes:
- If guest is not in a hung state, 'virsh destroy' does not drop the
secondary disk.
- 'virsh reset guest' does not exhibit the same behavior when guest is hung
- vdb remains attached.
- No logs produced in /var/log/libvirt indicating an issue, even with debug
logging enabled.

Is there any obvious I'm missing? Is this expected behavior?

Guest XML via 'virsh dumpxml guest':
<domain type='kvm' id='17'>
  <name>guest-with-secondary-disk-01</name>
  <uuid>ab578b51-3afd-4d01-86ab-a16b1b953919</uuid>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type>
    <boot dev='hd'/>
    <bootmenu enable='yes'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source
file='/vm/guest-with-secondary-disk-01/guest-with-secondary-disk-01.img'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04'
function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source
file='/vm/guest-with-secondary-disk-01/guest-with-secondary-disk-01_2.img'/>
      <backingStore/>
      <target dev='vdb' bus='virtio'/>
      <alias name='virtio-disk1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08'
function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='piix3-uhci'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='01:02:03:99:99:99'/>
      <source bridge='bridge_99'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='eth0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05'
function='0x0'/>
    </interface>
    <channel type='unix'>
      <source mode='bind'
path='/var/lib/libvirt/qemu/guest-with-secondary-disk-01.agent'/>
      <target type='virtio' name='org.qemu.guest_agent.0'
state='connected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'>
      <alias name='input0'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input1'/>
    </input>
    <graphics type='vnc' port='5900' autoport='yes' listen='127.0.0.1'
keymap='en-us'>
      <listen type='address' address='127.0.0.1'/>
    </graphics>
    <video>
      <model type='cirrus' vram='16384' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
function='0x0'/>
    </memballoon>
    <rng model='virtio'>
      <backend model='random'>/dev/random</backend>
      <alias name='rng0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07'
function='0x0'/>
    </rng>
  </devices>
  <seclabel type='dynamic' model='selinux' relabel='yes'>
    <label>system_u:system_r:svirt_t:s0:c259,c495</label>
    <imagelabel>system_u:object_r:svirt_image_t:s0:c259,c495</imagelabel>
  </seclabel>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+107:+107</label>
    <imagelabel>+107:+107</imagelabel>
  </seclabel>
</domain>

Secondary disk XML:
<disk type='file' device='disk'>
  <driver name='qemu' type='qcow2' cache='none'/>
  <source
file='/vm/guest-with-secondary-disk-01/guest-with-secondary-disk-01_2.img'/>
  <target dev='vdb'/>
</disk>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/libvirt-users/attachments/20180911/ab71de96/attachment.htm>