[vfio-users] Kernel panic at vfio_intx_handler leads to low performance in guest VM
tyng at cock.li
tyng at cock.li
Sat Oct 21 05:17:17 UTC 2017
>> As I tested, `nointxmask=1` may cause a new error
>> "vfio: Error: Failed to setup INTx fd: Device or resource busy"
>> when assign GPU and onboard audio together,
>> This error was mentioned in
>> https://www.redhat.com/archives/vfio-users/2016-March/msg00035.html
> Not surprising with all those other devices on the same interrupt.
> nointxmask requires that the interrupt is exclusive on the host.
> The log below just shows that vfio-pci can register both
> 00:1f.3 and 01:00.0 with nointxmask at the same time because
> they share an interrupt.
i am getting the same error
how can i passthrough two pci devices if they share an interrupt?
what i tried:
intel_iommu=on pcie_acs_override=downstream
intel_iommu=on pcie_acs_override=id=01:00.0,01:00.1,07:04.0
iommu: Adding device 0000:00:00.0 to group 0
iommu: Adding device 0000:00:01.0 to group 1
iommu: Adding device 0000:00:02.0 to group 2
iommu: Adding device 0000:00:03.0 to group 3
iommu: Adding device 0000:00:14.0 to group 4
iommu: Adding device 0000:00:16.0 to group 5
iommu: Adding device 0000:00:1b.0 to group 6
iommu: Adding device 0000:00:1c.0 to group 7
iommu: Adding device 0000:00:1c.2 to group 8
iommu: Adding device 0000:00:1c.3 to group 9
iommu: Adding device 0000:00:1c.4 to group 10
iommu: Adding device 0000:00:1f.0 to group 11
iommu: Adding device 0000:00:1f.2 to group 11
iommu: Adding device 0000:00:1f.3 to group 11
iommu: Adding device 0000:01:00.0 to group 1
iommu: Adding device 0000:01:00.1 to group 1
iommu: Adding device 0000:03:00.0 to group 12
iommu: Adding device 0000:05:00.0 to group 13
iommu: Adding device 0000:06:00.0 to group 14
iommu: Adding device 0000:07:04.0 to group 14
/sys/kernel/iommu_groups/0/devices/0000:00:00.0 ->
../../../../devices/pci0000:00/0000:00:00.0
/sys/kernel/iommu_groups/1/devices/0000:00:01.0 ->
../../../../devices/pci0000:00/0000:00:01.0
/sys/kernel/iommu_groups/1/devices/0000:01:00.0 ->
../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0
/sys/kernel/iommu_groups/1/devices/0000:01:00.1 ->
../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1
/sys/kernel/iommu_groups/2/devices/0000:00:02.0 ->
../../../../devices/pci0000:00/0000:00:02.0
/sys/kernel/iommu_groups/3/devices/0000:00:03.0 ->
../../../../devices/pci0000:00/0000:00:03.0
/sys/kernel/iommu_groups/4/devices/0000:00:14.0 ->
../../../../devices/pci0000:00/0000:00:14.0
/sys/kernel/iommu_groups/5/devices/0000:00:16.0 ->
../../../../devices/pci0000:00/0000:00:16.0
/sys/kernel/iommu_groups/6/devices/0000:00:1b.0 ->
../../../../devices/pci0000:00/0000:00:1b.0
/sys/kernel/iommu_groups/7/devices/0000:00:1c.0 ->
../../../../devices/pci0000:00/0000:00:1c.0
/sys/kernel/iommu_groups/8/devices/0000:00:1c.2 ->
../../../../devices/pci0000:00/0000:00:1c.2
/sys/kernel/iommu_groups/9/devices/0000:00:1c.3 ->
../../../../devices/pci0000:00/0000:00:1c.3
/sys/kernel/iommu_groups/10/devices/0000:00:1c.4 ->
../../../../devices/pci0000:00/0000:00:1c.4
/sys/kernel/iommu_groups/11/devices/0000:00:1f.0 ->
../../../../devices/pci0000:00/0000:00:1f.0
/sys/kernel/iommu_groups/11/devices/0000:00:1f.2 ->
../../../../devices/pci0000:00/0000:00:1f.2
/sys/kernel/iommu_groups/11/devices/0000:00:1f.3 ->
../../../../devices/pci0000:00/0000:00:1f.3
/sys/kernel/iommu_groups/12/devices/0000:03:00.0 ->
../../../../devices/pci0000:00/0000:00:1c.2/0000:03:00.0
/sys/kernel/iommu_groups/13/devices/0000:05:00.0 ->
../../../../devices/pci0000:00/0000:00:1c.3/0000:05:00.0
/sys/kernel/iommu_groups/14/devices/0000:06:00.0 ->
../../../../devices/pci0000:00/0000:00:1c.4/0000:06:00.0
/sys/kernel/iommu_groups/14/devices/0000:07:04.0 ->
../../../../devices/pci0000:00/0000:00:1c.4/0000:06:00.0/0000:07:04.0
options vfio-pci ids=10de:13c2,10de:0fbb,13f6:8788
blacklist nouveau
blacklist snd_hda_intel
blacklist snd_virtuoso
/sys/bus/pci/drivers/vfio-pci/0000:01:00.0 ->
../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0
/sys/bus/pci/drivers/vfio-pci/0000:01:00.1 ->
../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1
/sys/bus/pci/drivers/vfio-pci/0000:07:04.0 ->
../../../../devices/pci0000:00/0000:00:1c.4/0000:06:00.0/0000:07:04.0
00:00.0 Host bridge [0600]: Intel Corporation 4th Gen Core Processor
DRAM Controller [8086:0c00] (rev 06)
00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen
Core Processor PCI Express x16 Controller [8086:0c01] (rev 06)
00:02.0 VGA compatible controller [0300]: Intel Corporation Xeon E3-1200
v3/4th Gen Core Processor Integrated Graphics Controller [8086:0412]
(rev 06)
00:03.0 Audio device [0403]: Intel Corporation Xeon E3-1200 v3/4th Gen
Core Processor HD Audio Controller [8086:0c0c] (rev 06)
00:14.0 USB controller [0c03]: Intel Corporation 9 Series Chipset Family
USB xHCI Controller [8086:8cb1]
00:16.0 Communication controller [0780]: Intel Corporation 9 Series
Chipset Family ME Interface #1 [8086:8cba]
00:1b.0 Audio device [0403]: Intel Corporation 9 Series Chipset Family
HD Audio Controller [8086:8ca0]
00:1c.0 PCI bridge [0604]: Intel Corporation 9 Series Chipset Family PCI
Express Root Port 1 [8086:8c90] (rev d0)
00:1c.2 PCI bridge [0604]: Intel Corporation 9 Series Chipset Family PCI
Express Root Port 3 [8086:8c94] (rev d0)
00:1c.3 PCI bridge [0604]: Intel Corporation 9 Series Chipset Family PCI
Express Root Port 4 [8086:8c96] (rev d0)
00:1c.4 PCI bridge [0604]: Intel Corporation 9 Series Chipset Family PCI
Express Root Port 5 [8086:8c98] (rev d0)
00:1f.0 ISA bridge [0601]: Intel Corporation 9 Series Chipset Family Z97
LPC Controller [8086:8cc4]
00:1f.2 SATA controller [0106]: Intel Corporation 9 Series Chipset
Family SATA Controller [AHCI Mode] [8086:8c82]
00:1f.3 SMBus [0c05]: Intel Corporation 9 Series Chipset Family SMBus
Controller [8086:8ca2]
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204
[GeForce GTX 970] [10de:13c2] (rev a1)
Subsystem: ASUSTeK Computer Inc. GM204 [GeForce GTX 970]
[1043:8508]
Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Interrupt: pin A routed to IRQ 16
Region 0: Memory at f6000000 (32-bit, non-prefetchable)
[size=16M]
Region 1: Memory at e0000000 (64-bit, prefetchable) [size=256M]
Region 3: Memory at f0000000 (64-bit, prefetchable) [size=32M]
Region 5: I/O ports at e000 [size=128]
Expansion ROM at f7000000 [disabled] [size=512K]
Capabilities: [60] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D3 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [78] Express (v2) Legacy Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
unlimited, L1 <64us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1,
Exit Latency L0s <1us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x16, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range AB, TimeoutDis+,
LTR+, OBFF Via message
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
LTR+, OBFF Disabled
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance-
SpeedDis-
Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB,
EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2-, EqualizationPhase3-,
LinkEqualizationRequest-
Capabilities: [100 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1
RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128-
WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=01
Status: NegoPending- InProgress-
Capabilities: [250 v1] Latency Tolerance Reporting
Max snoop latency: 0ns
Max no snoop latency: 0ns
Capabilities: [258 v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+
ASPM_L1.1+ L1_PM_Substates+
PortCommonModeRestoreTime=255us
PortTPowerOnTime=10us
L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2-
ASPM_L1.1-
T_CommonMode=0us LTR1.2_Threshold=0ns
L1SubCtl2: T_PwrOn=10us
Capabilities: [128 v1] Power Budgeting <?>
Capabilities: [600 v1] Vendor Specific Information: ID=0001
Rev=1 Len=024 <?>
Capabilities: [900 v1] #19
Kernel driver in use: vfio-pci
Kernel modules: nouveau
01:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition
Audio Controller [10de:0fbb] (rev a1)
Subsystem: ASUSTeK Computer Inc. GM204 High Definition Audio
Controller [1043:8508]
Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Interrupt: pin B routed to IRQ 17
Region 0: Memory at f7080000 (32-bit, non-prefetchable)
[size=16K]
Capabilities: [60] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D3 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [78] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
unlimited, L1 <64us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
SlotPowerLimit 75.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1,
Exit Latency L0s <1us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x16, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range AB, TimeoutDis+,
LTR+, OBFF Via message
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
LTR-, OBFF Disabled
LnkSta2: Current De-emphasis Level: -3.5dB,
EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-,
LinkEqualizationRequest-
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel
03:00.0 PCI bridge [0604]: Intel Corporation 82801 PCI Bridge
[8086:244e] (rev 41)
05:00.0 Ethernet controller [0200]: Qualcomm Atheros Killer E220x
Gigabit Ethernet Controller [1969:e091] (rev 10)
06:00.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1083/1085 PCIe to
PCI Bridge [1b21:1080] (rev 04)
07:04.0 Multimedia audio controller [0401]: C-Media Electronics Inc
CMI8788 [Oxygen HD Audio] [13f6:8788]
Subsystem: ASUSTeK Computer Inc. Virtuoso 100 (Xonar Essence
STX) [1043:835c]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Interrupt: pin A routed to IRQ 16
Region 0: I/O ports at c000 [disabled] [size=256]
Capabilities: [c0] Power Management version 2
Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D3 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Kernel driver in use: vfio-pci
Kernel modules: snd_virtuoso
$ qemu-system-x86_64 -device vfio-pci,host=01:00.0 -device
vfio-pci,host=01:00.1
vfio_ecap_init: 0000:01:00.0 hiding ecap 0x1e at 0x258
vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19 at 0x900
$ qemu-system-x86_64 -device vfio-pci,host=07:04.0 -device
vfio-pci,host=01:00.1
vfio-pci 0000:07:04.0: enabling device (0000 -> 0001)
$ qemu-system-x86_64 -device vfio-pci,host=07:04.0 -device
vfio-pci,host=01:00.0
vfio error: 0000:07:04.0: failed to setup INTx fd: Operation not
permitted
vfio_ecap_init: 0000:01:00.0 hiding ecap 0x1e at 0x258
vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19 at 0x900
genirq: Flags mismatch irq 16. 00000000 (vfio-intx(0000:07:04.0)) vs.
00000080 (vfio-intx(0000:01:00.0))
More information about the vfio-users
mailing list