[vfio-users] PCI USB card duplicate vendor id issue

[Alex Williamson]

On Mon, 20 Nov 2017 17:44:12 +0000
Alex Kretzschmar <alexktz at gmail.com> wrote:

> I purchased a cheap £3 USB PCI card to fit into a Supermicro X11SAE-F.
> USB card is a Belkin 5 Port USB 2.0 PCI Card 4 Int / 1 Ext Ports from
> Scan.co.uk, item number LN46145. It uses an NEC chipset. The
> configuration of this card is proving tricky due to a duplicate
> assignment of device and vendors IDs as evidenced with lspci -nn.
> 
> 
> 05:01.0 USB controller [0c03]: NEC Corporation OHCI USB Controller
> [1033:0035] (rev 43)
> 05:01.1 USB controller [0c03]: NEC Corporation OHCI USB Controller
> [1033:0035] (rev 43)
> 05:01.2 USB controller [0c03]: NEC Corporation uPD72010x USB 2.0
> Controller [1033:00e0] (rev 04)
> 
> 1033:0035 specified from 05:01.0 and 05:01.1. If I add all of these to
> /etc/modprobe.d/vfio.conf alongside my GPU IDs then QEMU fails to
> start the guest complaining it can't attach to 05:01.1. The error is
> this:
> 
> Error starting domain: internal error: qemu unexpectedly closed the
> monitor: 2017-11-16T19:58:16.522974Z qemu-system-x86_64: -chardev
> pty,id=charserial0: char device redirected to /dev/pts/1 (label
> charserial0)
> 2017-11-16T19:58:17.917457Z qemu-system-x86_64: -device
> vfio-pci,host=05:01.1,id=hostdev3,bus=pci.0,addr=0xc: vfio error:
> 0000:05:01.1: failed to setup INTx fd: Operation not permitted
> 
> Traceback (most recent call last):
>   File "/usr/share/virt-manager/virtManager/asyncjob.py", line 89, in cb_wrapper
>     callback(asyncjob, *args, **kwargs)
>   File "/usr/share/virt-manager/virtManager/asyncjob.py", line 125, in tmpcb
>     callback(*args, **kwargs)
>   File "/usr/share/virt-manager/virtManager/libvirtobject.py", line 82, in newfn
>     ret = fn(self, *args, **kwargs)
>   File "/usr/share/virt-manager/virtManager/domain.py", line 1505, in startup
>     self._backend.create()
>   File "/usr/lib/python2.7/site-packages/libvirt.py", line 1069, in create
>     if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
> libvirtError: internal error: qemu unexpectedly closed the monitor:
> 2017-11-16T19:58:16.522974Z qemu-system-x86_64: -chardev
> pty,id=charserial0: char device redirected to /dev/pts/1 (label
> charserial0)
> 2017-11-16T19:58:17.917457Z qemu-system-x86_64: -device
> vfio-pci,host=05:01.1,id=hostdev3,bus=pci.0,addr=0xc: vfio error:
> 0000:05:01.1: failed to setup INTx fd: Operation not permitted
> 
> My host OS is Arch running the linux-vfio kernel for ACS support on an
> i7 6700k. vt-d is otherwise working perfectly.
> 
> The devices are not being grabbed by vfio-pci as I would hope at boot.
> 
> 
> 05:01.0 USB controller: NEC Corporation OHCI USB Controller (rev 43)
>     Subsystem: NEC Corporation USB Controller
>     Kernel driver in use: ohci-pci
>     Kernel modules: ohci_pci
> 05:01.1 USB controller: NEC Corporation OHCI USB Controller (rev 43)
>     Subsystem: NEC Corporation USB Controller
>     Kernel driver in use: ohci-pci
>     Kernel modules: ohci_pci
> 05:01.2 USB controller: NEC Corporation uPD72010x USB 2.0 Controller (rev 04)
>     Subsystem: Device 0ee4:3383
>     Kernel driver in use: ehci-pci
>     Kernel modules: ehci_pci
> 
> Following some help I received elsewhere I tried the manual approach
> which did successfully bind the devices to vfio-pci but I still ended
> up with the guest not starting due to the IRQ conflict below:
> 
> [   93.950195] vfio-pci 0000:05:01.0: enabling device (0000 -> 0002)
> [   94.000193] vfio-pci 0000:05:01.2: enabling device (0000 -> 0002)
> [   94.034477] genirq: Flags mismatch irq 17. 00000000
> (vfio-intx(0000:05:01.2)) vs. 00000080 (idma64.1)
> 
> Manual bind steps followed:
> 
> 
> echo 0000:05:01.0 > /sys/bus/pci/devices/0000:05:01.0/driver/unbind
> echo 0000:05:01.1 > /sys/bus/pci/devices/0000:05:01.1/driver/unbind
> echo 0000:05:01.2 > /sys/bus/pci/devices/0000:05:01.2/driver/unbind
> 
> echo 1033 0035 > /sys/bus/pci/drivers/vfio-pci/new_id
> echo 1033 00e0 > /sys/bus/pci/drivers/vfio-pci/new_id
> 
> echo 0000:05:01.0 > /sys/bus/pci/drivers/vfio-pci/bind
> echo 0000:05:01.1 > /sys/bus/pci/drivers/vfio-pci/bind
> echo 0000:05:01.2 > /sys/bus/pci/drivers/vfio-pci/bind
> 
> # lspci -k
> 
> 05:01.0 USB controller: NEC Corporation OHCI USB Controller (rev 43)
> Subsystem: NEC Corporation USB Controller
> Kernel driver in use: vfio-pci
> Kernel modules: ohci_pci
> 
> At this point I'm fairly certain the card is garbage and unsalvageable, is it?

The irq mismatch indicates the card is getting probed as not supporting
INTx masking, which means that it requires an exclusive interrupt in
the host, but another device identified as idma64.1 is already using
the interrupt line.  You might be able to change slots, but AIUI there
are numerous grey market NEC clones around and all of them are junk.
Given the price, seems like a fair chance you picked up one of those.
Be leery that this may not be the only way it acts up.  Thanks,

Alex