[vfio-users] USB card prevents qemu finding its harddrive

[Ede Wolf]

So I've bought this rather expensive USB 3.0 controller, making sure it 
was compatible to everything on this planet and having a native pcie v2 
interface and now I am having the problem, as soon as it gets passed 
through, qemu cannot find it's boot disk any more.

"Boot failed. Could not read the boot disk". The boot menu only shows 
legacy option rom or pxe.

I have no remote idea, where the relation between those two could be. It 
is not, that the start up of the VM itself is failing, because of a 
misconfiguration, it's just, that after powering on of the guest, it 
cannot find any harddrives any more and tries to pxe boot instead.

As soon as I remove:
-device vfio-pci,host=0a:00.0,x-vga=off

from the configuration, the guest starts just normal - with another card 
still passed through. So it is not a general passthrough problem. And 
I've had passed through two cards before.

Needless to say, the guest image lives on a regular sata harddrive and 
is not attached to that card in any way.
In addition, it is the only device in its vfio group, making it an even 
more simple config:

# journalctl -k | grep -i iommu | grep 25
Dez 22 11:50:17 kernel: pci 0000:0a:00.0: Adding to iommu group 25

# lspci -s "0a:00.0" -n -v
0a:00.0 0c03: 1106:3483 (rev 01) (prog-if 30 [XHCI])
         Subsystem: 1106:3483
         Flags: fast devsel, IRQ 16, NUMA node 0
         Memory at bf700000 (64-bit, non-prefetchable) [size=4K]
         Capabilities: [80] Power Management version 3
         Capabilities: [90] MSI: Enable- Count=1/4 Maskable- 64bit+
         Capabilities: [c4] Express Endpoint, MSI 00
         Capabilities: [100] Advanced Error Reporting
         Kernel driver in use: vfio-pci
         Kernel modules: xhci_pci


How is it possible that this card affects the boot drive of qemu? Where 
is the connection between "-drive file=/home/VM/guest.img" and "-device 
vfio-pci"? And how to eventually debug this?

Since this behaviour is related to vfio-pci, I am hoping, this is still 
the correct list, even though this seems to affect qemu as whole. In a 
quite obscure, magical way, one might add.


Thanks very much for any help

Ede


P.S.: Preparation, even though the config should not be the issue:

# The Delock 5x USB Card (VIA)
ExecStart=-/usr/bin/sh -c 'echo "0000:0a:00.0" > 
/sys/bus/pci/devices/0000:0a:00.0/driver/unbind'
ExecStart=/usr/bin/sh -c 'echo 1106 3483 > 
/sys/bus/pci/drivers/vfio-pci/new_id'
ExecStart=/usr/bin/sh -c 'chgrp kvm /dev/vfio/25'
ExecStart=/usr/bin/sh -c 'chmod 0660 /dev/vfio/25'