[vfio-users] Passthrough of Thunderbolt SATA Controller

Fri Jun 11 02:04:28 UTC 2021

Adding that I got the device to pass, but there are errors.

To start with, I found a post (including Alex Williamson's help) about
finding the NHI device and binding that vfio.  This approach worked.
Currently my script to detach the devices and bind them to vfio is:

#!/bin/bash

#NHI device
virsh nodedev-detach pci_0000_00_0d_2

#OWC devices
#SATA controllers
virsh nodedev-detach pci_0000_03_00_0
virsh nodedev-detach pci_0000_04_00_0
#PCI Tbolt devices
virsh nodedev-detach pci_0000_01_00_0
virsh nodedev-detach pci_0000_02_01_0
virsh nodedev-detach pci_0000_02_02_0

Followed by qemu:
sudo qemu-system-x86_64 \
-vnc :0 \
-m 2048 -smp cores=2 \
-machine accel=kvm \
-drive
file=/usr/share/OVMF/x64/OVMF_CODE.fd,if=pflash,format=raw,unit=0,readonly=on
\
-drive file=/home/chuwi-admin/OVMF_VARS.fd,if=pflash,format=raw,unit=1 \
-drive format=raw,file=/dev/sdb \
-device vfio-pci,host=03:00.0 \
-device vfio-pci,host=04:00.0

Now inside the VM running an Arch USB install image, the drives are
detected.  I am having trouble getting access to them.  The initial dd
/dev/zero to one of the disk devices errored out with timeouts and finally
said it is a read only device.

failed command: READ FPDMA QUEUED

also

revalidation failed

Anytime I try to mount a disk, make a new fs on a disk, or basically do
anything there are I/O errors.

On Thu, Jun 10, 2021 at 5:59 PM kram threetwoone <kram321 at gmail.com> wrote:

> Hello everyone, and thanks for everything you do!
>
> The device is a OWC thunderbay 4.
> https://www.owcdigital.com/products/thunderbay-4
> Inside it contains two SATA controllers ASM1062, and one Tbolt 3 Intel
> DSL6540.
>
> The host is a Chuwi Corebox pro.
> https://store.chuwi.com/products/corebox-pro
>
> I'll say I'm pretty familiar with GPU passthrough and have been doing it
> since Oct 2014.  The idea is to passthrough this OWC box to a headless
> server via qemu/libvirt (no virt-manager).  So far I have tried many
> techniques such as:
>
> virsh nodedev-detach
> manual modprobe vfio-pci
> booting kernel with module vfio-pci ids=8086:15d3 (thunderbolt) and 1b21:1060
> (SATA controller)
>
> No matter what I do, the only success I have reached is having the Intel
> controller show up without a driver in use on lspci -nnk, but the vfio
> device /dev/vfio/XXX never get created.
>
> Please help, I'm new to passing through non-GPU devices.  Thanks in
> advance.  For what it's worth, this exact box I have been able to
> passthrough the Intel integrated GPU, so vfio is working.
>
> Any further information can be provided as requested, this is already too
> much text.
>
> All following logs are generated as a stock system, everything reset to a
> vanilla machine (no blacklists or vfio module loaded at boot):
> LSPCI -nnk:
> 00:00.0 Host bridge [0600]: Intel Corporation Device [8086:8a02] (rev 03)
>        Subsystem: Intel Corporation Device [8086:7270]
>        Kernel driver in use: icl_uncore
> 00:02.0 VGA compatible controller [0300]: Intel Corporation Iris Plus
> Graphics G1 (Ice Lake) [8086:8a56] (rev 07)
>        Subsystem: Device [1e50:800a]
>        Kernel driver in use: i915
>        Kernel modules: i915
> 00:07.0 PCI bridge [0604]: Intel Corporation Ice Lake Thunderbolt 3 PCI
> Express Root Port #0 [8086:8a1d] (rev 03)
>        Kernel driver in use: pcieport
> 00:0d.0 USB controller [0c03]: Intel Corporation Ice Lake Thunderbolt 3
> USB Controller [8086:8a13] (rev 03)
>        Subsystem: Intel Corporation Device [8086:7270]
>        Kernel driver in use: xhci_hcd
>        Kernel modules: xhci_pci
> 00:0d.2 System peripheral [0880]: Intel Corporation Ice Lake Thunderbolt 3
> NHI #0 [8086:8a17] (rev 03)
>        Kernel driver in use: thunderbolt
>        Kernel modules: thunderbolt
> 00:14.0 USB controller [0c03]: Intel Corporation Ice Lake-LP USB 3.1 xHCI
> Host Controller [8086:34ed] (rev 30)
>        Subsystem: Intel Corporation Device [8086:7270]
>        Kernel driver in use: xhci_hcd
>        Kernel modules: xhci_pci
> 00:14.2 RAM memory [0500]: Intel Corporation Device [8086:34ef] (rev 30)
>        Subsystem: Intel Corporation Device [8086:7270]
> 00:14.3 Network controller [0280]: Intel Corporation Killer Wi-Fi 6
> AX1650i 160MHz Wireless Network Adapter (201NGW) [8086:34f0] (rev 30)
>        Subsystem: Intel Corporation Device [8086:0074]
>        Kernel driver in use: iwlwifi
>        Kernel modules: iwlwifi
> 00:15.0 Serial bus controller [0c80]: Intel Corporation Ice Lake-LP Serial
> IO I2C Controller #0 [8086:34e8] (rev 30)
>        Subsystem: Intel Corporation Device [8086:7270]
>        Kernel driver in use: intel-lpss
>        Kernel modules: intel_lpss_pci
> 00:16.0 Communication controller [0780]: Intel Corporation Management
> Engine Interface [8086:34e0] (rev 30)
>        Subsystem: Intel Corporation Device [8086:7270]
>        Kernel driver in use: mei_me
>        Kernel modules: mei_me
> 00:17.0 SATA controller [0106]: Intel Corporation Ice Lake-LP SATA
> Controller [AHCI mode] [8086:34d3] (rev 30)
>        Subsystem: Intel Corporation Device [8086:7270]
>        Kernel driver in use: ahci
> 00:1c.0 PCI bridge [0604]: Intel Corporation Device [8086:34be] (rev 30)
>        Kernel driver in use: pcieport
> 00:1c.7 PCI bridge [0604]: Intel Corporation Device [8086:34bf] (rev 30)
>        Kernel driver in use: pcieport
> 00:1d.0 PCI bridge [0604]: Intel Corporation Device [8086:34b4] (rev 30)
>        Kernel driver in use: pcieport
> 00:1e.0 Communication controller [0780]: Intel Corporation Ice Lake-LP
> Serial IO UART Controller #0 [8086:34a8] (rev 30)
>        Subsystem: Intel Corporation Device [8086:7270]
>        Kernel driver in use: intel-lpss
>        Kernel modules: intel_lpss_pci
> 00:1e.3 Serial bus controller [0c80]: Intel Corporation Ice Lake-LP Serial
> IO SPI Controller #1 [8086:34ab] (rev 30)
>        Subsystem: Intel Corporation Device [8086:7270]
>        Kernel driver in use: intel-lpss
>        Kernel modules: intel_lpss_pci
> 00:1f.0 ISA bridge [0601]: Intel Corporation Ice Lake-LP LPC Controller
> [8086:3482] (rev 30)
>        Subsystem: Intel Corporation Device [8086:7270]
> 00:1f.3 Audio device [0403]: Intel Corporation Smart Sound Technology
> Audio Controller [8086:34c8] (rev 30)
>        Subsystem: Realtek Semiconductor Co., Ltd. Device [10ec:119e]
>        Kernel driver in use: snd_hda_intel
>        Kernel modules: snd_hda_intel, snd_sof_pci_intel_icl
> 00:1f.4 SMBus [0c05]: Intel Corporation Ice Lake-LP SMBus Controller
> [8086:34a3] (rev 30)
>        Subsystem: Intel Corporation Device [8086:7270]
>        Kernel driver in use: i801_smbus
>        Kernel modules: i2c_i801
> 00:1f.5 Serial bus controller [0c80]: Intel Corporation Ice Lake-LP SPI
> Controller [8086:34a4] (rev 30)
>        Subsystem: Intel Corporation Device [8086:7270]
> 01:00.0 PCI bridge [0604]: Intel Corporation JHL6540 Thunderbolt 3 Bridge
> (C step) [Alpine Ridge 4C 2016] [8086:15d3] (rev 02)
>        Kernel driver in use: pcieport
> 02:01.0 PCI bridge [0604]: Intel Corporation JHL6540 Thunderbolt 3 Bridge
> (C step) [Alpine Ridge 4C 2016] [8086:15d3] (rev 02)
>        Kernel driver in use: pcieport
> 02:02.0 PCI bridge [0604]: Intel Corporation JHL6540 Thunderbolt 3 Bridge
> (C step) [Alpine Ridge 4C 2016] [8086:15d3] (rev 02)
>        Kernel driver in use: pcieport
> 02:04.0 PCI bridge [0604]: Intel Corporation JHL6540 Thunderbolt 3 Bridge
> (C step) [Alpine Ridge 4C 2016] [8086:15d3] (rev 02)
>        Kernel driver in use: pcieport
> 03:00.0 SATA controller [0106]: ASMedia Technology Inc. Device [1b21:0625]
> (rev 01)
>        Subsystem: ASMedia Technology Inc. Device [1b21:1060]
>        Kernel driver in use: ahci
> 04:00.0 SATA controller [0106]: ASMedia Technology Inc. Device [1b21:0625]
> (rev 01)
>        Subsystem: ASMedia Technology Inc. Device [1b21:1060]
>        Kernel driver in use: ahci
> 2c:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd.
> RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev
> 15)
>        Subsystem: Realtek Semiconductor Co., Ltd. Device [10ec:0123]
>        Kernel driver in use: r8169
>        Kernel modules: r8169
> 2d:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd.
> RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev
> 15)
>        Subsystem: Realtek Semiconductor Co., Ltd. Device [10ec:0123]
>        Kernel driver in use: r8169
>        Kernel modules: r8169
> 2e:00.0 Non-Volatile memory controller [0108]: Kingston Technology
> Company, Inc. Device [2646:5008] (rev 01)
>        Subsystem: Kingston Technology Company, Inc. Device [2646:5008]
>        Kernel driver in use: nvme
>
> IOMMU groups:
> IOMMU Group 0:
>        00:00.0 Host bridge [0600]: Intel Corporation Device [8086:8a02]
> (rev 03)
> IOMMU Group 1:
>        00:02.0 VGA compatible controller [0300]: Intel Corporation Iris
> Plus Graphics G1 (Ice Lake) [8086:8a56] (rev 07)
> IOMMU Group 2:
>        00:07.0 PCI bridge [0604]: Intel Corporation Ice Lake Thunderbolt 3
> PCI Express Root Port #0 [8086:8a1d] (rev 03)
> IOMMU Group 3:
>        00:0d.0 USB controller [0c03]: Intel Corporation Ice Lake
> Thunderbolt 3 USB Controller [8086:8a13] (rev 03)
>        00:0d.2 System peripheral [0880]: Intel Corporation Ice Lake
> Thunderbolt 3 NHI #0 [8086:8a17] (rev 03)
> IOMMU Group 4:
>        00:14.0 USB controller [0c03]: Intel Corporation Ice Lake-LP USB
> 3.1 xHCI Host Controller [8086:34ed] (rev 30)
>        00:14.2 RAM memory [0500]: Intel Corporation Device [8086:34ef]
> (rev 30)
> IOMMU Group 5:
>        00:14.3 Network controller [0280]: Intel Corporation Killer Wi-Fi 6
> AX1650i 160MHz Wireless Network Adapter (201NGW) [8086:34f0] (rev 30)
> IOMMU Group 6:
>        00:15.0 Serial bus controller [0c80]: Intel Corporation Ice Lake-LP
> Serial IO I2C Controller #0 [8086:34e8] (rev 30)
> IOMMU Group 7:
>        00:16.0 Communication controller [0780]: Intel Corporation
> Management Engine Interface [8086:34e0] (rev 30)
> IOMMU Group 8:
>        00:17.0 SATA controller [0106]: Intel Corporation Ice Lake-LP SATA
> Controller [AHCI mode] [8086:34d3] (rev 30)
> IOMMU Group 9:
>        00:1c.0 PCI bridge [0604]: Intel Corporation Device [8086:34be]
> (rev 30)
> IOMMU Group 10:
>        00:1c.7 PCI bridge [0604]: Intel Corporation Device [8086:34bf]
> (rev 30)
> IOMMU Group 11:
>        00:1d.0 PCI bridge [0604]: Intel Corporation Device [8086:34b4]
> (rev 30)
> IOMMU Group 12:
>        00:1e.0 Communication controller [0780]: Intel Corporation Ice
> Lake-LP Serial IO UART Controller #0 [8086:34a8] (rev 30)
>        00:1e.3 Serial bus controller [0c80]: Intel Corporation Ice Lake-LP
> Serial IO SPI Controller #1 [8086:34ab] (rev 30)
> IOMMU Group 13:
>        00:1f.0 ISA bridge [0601]: Intel Corporation Ice Lake-LP LPC
> Controller [8086:3482] (rev 30)
>        00:1f.3 Audio device [0403]: Intel Corporation Smart Sound
> Technology Audio Controller [8086:34c8] (rev 30)
>        00:1f.4 SMBus [0c05]: Intel Corporation Ice Lake-LP SMBus
> Controller [8086:34a3] (rev 30)
>        00:1f.5 Serial bus controller [0c80]: Intel Corporation Ice Lake-LP
> SPI Controller [8086:34a4] (rev 30)
> IOMMU Group 14:
>        01:00.0 PCI bridge [0604]: Intel Corporation JHL6540 Thunderbolt 3
> Bridge (C step) [Alpine Ridge 4C 2016] [8086:15d3] (rev 02)
> IOMMU Group 15:
>        02:01.0 PCI bridge [0604]: Intel Corporation JHL6540 Thunderbolt 3
> Bridge (C step) [Alpine Ridge 4C 2016] [8086:15d3] (rev 02)
>        03:00.0 SATA controller [0106]: ASMedia Technology Inc. Device
> [1b21:0625] (rev 01)
> IOMMU Group 16:
>        02:02.0 PCI bridge [0604]: Intel Corporation JHL6540 Thunderbolt 3
> Bridge (C step) [Alpine Ridge 4C 2016] [8086:15d3] (rev 02)
>        04:00.0 SATA controller [0106]: ASMedia Technology Inc. Device
> [1b21:0625] (rev 01)
> IOMMU Group 17:
>        02:04.0 PCI bridge [0604]: Intel Corporation JHL6540 Thunderbolt 3
> Bridge (C step) [Alpine Ridge 4C 2016] [8086:15d3] (rev 02)
> IOMMU Group 18:
>        2c:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd.
> RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev
> 15)
> IOMMU Group 19:
>        2d:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd.
> RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev
> 15)
> IOMMU Group 20:
>        2e:00.0 Non-Volatile memory controller [0108]: Kingston Technology
> Company, Inc. Device [2646:5008] (rev 01)
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20210610/a90d7986/attachment.htm>