[vfio-users] USB card prevents qemu finding its harddrive

[leesteken at pm.me]

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Wednesday, December 25, 2019 12:38 PM, Ede Wolf <listac at nebelschwaden.de> wrote:

> Hello,
>
> thanks for you answer. Unfortunately it does not seem to resolve this
> issue.
>
> Because, if I use if=ide instead of if=virtio, I do not have this issue.
>
> This holds true for both a 4.19 lts kernel as well as a more recent 5.4.6.
>
> Further, when using if=virtio, no matter what address I use for the
> passed through cards, the the harddrives are always missing.
> And in case I do use an already used address, qemu startup failes with
> an error, that this address is already used by [net|vga...]
>
> Also, I do have one device, that works. If I exchange that device with
> the problematic one, using the same bus/address, the harddrives (virtio
> disk pci:00:04.0/00:05.0) are still missing.
>
> Works (parallel card):
> -device vfio-pci,host=02:00.0,x-vga=off,bus=pci.0,addr=0x6
>
> Does not work (changed from 0a to 08 due to changed slot):
> -device vfio-pci,host=08:00.0,x-vga=off,bus=pci.0,addr=0x6
>
> Tried all adresses from 0x1 to 0x9. Either startup of qemu failed or no
> bootable harddrives were found.
>
> So it does not matter wether I pass through both devices or just the
> problematic one, the harddrives are missing any way.
>
> In addition, it makes no difference, wether I specify one or f.e. three
> drives (files). As soon as I add the problematic card, all are gone,
> when using the virtio interface.
>
> Of course I may have made a mistake, so I am happy about suggestions or
> corrections.
>

I'm out of ideas, sorry. Maybe more details information will provide a hint. What are your IOMMU groups?
Can you boot the VM using some Linux LiveCD (using IDE) and show the output of lspci -k (in both cases)?
Please also providing the whole kvm command line used to start the VM (in both cases).

> Ede
>
> Am 22.12.19 um 13:41 schrieb leesteken at pm.me:
>
> > Hi,
> > Adding a device might shift the other (virtual) devices including the disk controller, or the disk controller might conflict with the default address that is used when you use -device to add it, which might remove (or not add) the disk controller.
> > Try adding ,addr=0x?? to the -device parameter where you select an unused address for the ??.
> > You might have to select a (virtual) bus as well as part of the -device parameter.
> > hope this helps, Arjen
> > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> > On Sunday, December 22, 2019 1:14 PM, Ede Wolf listac at nebelschwaden.de wrote:
> >
> > > So I've bought this rather expensive USB 3.0 controller, making sure it
> > > was compatible to everything on this planet and having a native pcie v2
> > > interface and now I am having the problem, as soon as it gets passed
> > > through, qemu cannot find it's boot disk any more.
> > > "Boot failed. Could not read the boot disk". The boot menu only shows
> > > legacy option rom or pxe.
> > > I have no remote idea, where the relation between those two could be. It
> > > is not, that the start up of the VM itself is failing, because of a
> > > misconfiguration, it's just, that after powering on of the guest, it
> > > cannot find any harddrives any more and tries to pxe boot instead.
> > > As soon as I remove:
> > > -device vfio-pci,host=0a:00.0,x-vga=off
> > > from the configuration, the guest starts just normal - with another card
> > > still passed through. So it is not a general passthrough problem. And
> > > I've had passed through two cards before.
> > > Needless to say, the guest image lives on a regular sata harddrive and
> > > is not attached to that card in any way.
> > > In addition, it is the only device in its vfio group, making it an even
> > > more simple config:
> > > journalctl -k | grep -i iommu | grep 25
> > > ========================================
> > > Dez 22 11:50:17 kernel: pci 0000:0a:00.0: Adding to iommu group 25
> > > lspci -s "0a:00.0" -n -v
> > > =========================
> > > 0a:00.0 0c03: 1106:3483 (rev 01) (prog-if 30 [XHCI])
> > > Subsystem: 1106:3483
> > > Flags: fast devsel, IRQ 16, NUMA node 0
> > > Memory at bf700000 (64-bit, non-prefetchable) [size=4K]
> > > Capabilities: [80] Power Management version 3
> > > Capabilities: [90] MSI: Enable- Count=1/4 Maskable- 64bit+
> > > Capabilities: [c4] Express Endpoint, MSI 00
> > > Capabilities: [100] Advanced Error Reporting
> > > Kernel driver in use: vfio-pci
> > > Kernel modules: xhci_pci
> > > How is it possible that this card affects the boot drive of qemu? Where
> > > is the connection between "-drive file=/home/VM/guest.img" and "-device
> > > vfio-pci"? And how to eventually debug this?
> > > Since this behaviour is related to vfio-pci, I am hoping, this is still
> > > the correct list, even though this seems to affect qemu as whole. In a
> > > quite obscure, magical way, one might add.
> > > Thanks very much for any help
> > > Ede
> > > P.S.: Preparation, even though the config should not be the issue:
> > > The Delock 5x USB Card (VIA)
> > > =============================
> > > ExecStart=-/usr/bin/sh -c 'echo "0000:0a:00.0" >
> > > /sys/bus/pci/devices/0000:0a:00.0/driver/unbind'
> > > ExecStart=/usr/bin/sh -c 'echo 1106 3483 >
> > > /sys/bus/pci/drivers/vfio-pci/new_id'
> > > ExecStart=/usr/bin/sh -c 'chgrp kvm /dev/vfio/25'
> > > ExecStart=/usr/bin/sh -c 'chmod 0660 /dev/vfio/25'
> > > vfio-users mailing list
> > > vfio-users at redhat.com
> > > https://www.redhat.com/mailman/listinfo/vfio-users
>
> vfio-users mailing list
> vfio-users at redhat.com
> https://www.redhat.com/mailman/listinfo/vfio-users