[vfio-users] "Device or Resource Busy"

[Bryan Jacobs]

Hello VFIO users!

Yesterday I was able to boot a Windows VM successfully with OVMF and a
passed-through graphics card.

I'm not sure what changed, but today I get a libvirt qemu log full of these
messages:

"qemu-system-x86_64: vfio_region_write(0000:01:00.0:region1+0x240c14,
0xffffffff,4) failed: Device or resource busy"

There are hundreds upon hundreds of these - it looks like it's walking a
large memory space, doing reads and writes, and failing every time. The
graphics card gives code 43, but the problem doesn't go away even if I
entirely disable enlightenments. Removing the PCIe passthrough devices gets
rid of the log entries.

My two passed PCI devices are a GTX 980TI GPU and its HDMI audio function.
They are in their own IOMMU group with their own pci bridge. The CPU is an
i5-6600k.

I have verified that the devices are bound to the vfio-pci driver, although
I don't seem to be able to entirely stop console output from appearing on
the graphics card on boot (not even with my CSM-free UEFI set to use the
integrated graphics instead). There's definitely nothing like X running,
and the nvidia module is unloaded in the host OS.

# virsh --version
1.3.2
# uname -a
Linux host 4.4.5-1-ARCH #1 SMP PREEMPT Thu Mar 10 07:38:19 CET 2016 x86_64
GNU/Linux

Removing the passed-through device fixes the issue.

What would make the vfio_region_(read|write) calls return EBUSY? I figured
I'd ask here before diving into the kernel sources :-/.

Libvirt domain XML:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>windows</name>
  <uuid>58e2af6c-eda7-4546-b3aa-53d05c174586</uuid>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <memoryBacking>
    <hugepages/>
  </memoryBacking>
  <vcpu placement='static'>3</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.5'>hvm</type>
    <loader readonly='yes'
type='pflash'>/usr/share/ovmf/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/windows_VARS.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <kvm>
      <hidden state='on'/>
    </kvm>
    <vmport state='off'/>
  </features>
  <cpu mode='host-model'>
    <model fallback='allow'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/sbin/qemu-system-x86_64</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/somevg/windows'/>
      <target dev='vdb' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04'
function='0x0'/>
    </disk>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='nec-xhci'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
function='0x0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a'
function='0x0'/>
    </controller>
    <interface type='direct'>
      <mac address='52:54:00:2a:c0:48'/>
      <source dev='enp0s31f6' mode='bridge'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07'
function='0x0'/>
    </interface>
    <console type='pty'>
      <target type='virtio' port='0'/>
    </console>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <input type='tablet' bus='usb'/>
    <graphics type='vnc' port='5900' autoport='no' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08'
function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09'
function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x28de'/>
        <product id='0x1142'/>
      </source>
    </hostdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05'
function='0x0'/>
    </memballoon>
    <rng model='virtio'>
      <backend model='random'>/dev/random</backend>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
function='0x0'/>
    </rng>
  </devices>
  <qemu:commandline>
    <qemu:arg value='-cpu'/>
    <qemu:arg
value='host,kvm=off,hv_relaxed,hv_spinlocks=0x1fff,hv_time,hv_vendor_id=whatever'/>
  </qemu:commandline>
</domain>

Thanks!

Bryan Jacobs
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20160316/9b8e630a/attachment.htm>