[vfio-users] ACS override not working (can not split IOMMU groups)
Péter Takács
takacs.peter90 at gmail.com
Sat Feb 23 21:00:01 UTC 2019
Hello!
I need some help to split my IOMMU groups, let’s start from the beginning.
I have an Asrock J3455-ITX motherboard with the latest Proxmox version. On
the motherboard there are an onboard NIC, an external NIC (plugged into the
only one PCIe slot) and an external Wifi card (plugged into the M.2 slot).
On the host proxmox I have a pfSense running in a VM and I would like to
pass through my external NIC and the Wifi card but they are in the same
IOMMU group than the onbord NIC.
See my dmesg and lspci output:
# dmesg |grep -i iommu
[ 0.000000] Command line:
BOOT_IMAGE=/ROOT/pve-1@/boot/vmlinuz-4.15.18-10-pve
root=ZFS=rpool/ROOT/pve-1 ro root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet
intel_iommu=on iommu=pt pcie_acs_override=downstream
[ 0.000000] Warning: PCIe ACS overrides enabled; This may allow
non-IOMMU protected peer-to-peer DMA
[ 0.000000] Kernel command line:
BOOT_IMAGE=/ROOT/pve-1@/boot/vmlinuz-4.15.18-10-pve
root=ZFS=rpool/ROOT/pve-1 ro root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet
intel_iommu=on iommu=pt pcie_acs_override=downstream
[ 0.000000] DMAR: IOMMU enabled
[ 0.004000] DMAR-IR: IOAPIC id 1 under DRHD base 0xfed65000 IOMMU 1
[ 1.340272] iommu: Adding device 0000:00:00.0 to group 0
[ 1.340287] iommu: Adding device 0000:00:02.0 to group 1
[ 1.340306] iommu: Adding device 0000:00:0e.0 to group 2
[ 1.340325] iommu: Adding device 0000:00:0f.0 to group 3
[ 1.340339] iommu: Adding device 0000:00:12.0 to group 4
[ 1.340384] iommu: Adding device 0000:00:13.0 to group 5
[ 1.340408] iommu: Adding device 0000:00:13.1 to group 5
[ 1.340427] iommu: Adding device 0000:00:13.2 to group 5
[ 1.340447] iommu: Adding device 0000:00:13.3 to group 5
[ 1.340465] iommu: Adding device 0000:00:15.0 to group 6
[ 1.340487] iommu: Adding device 0000:00:1f.0 to group 7
[ 1.340500] iommu: Adding device 0000:00:1f.1 to group 7
[ 1.340515] iommu: Adding device 0000:01:00.0 to group 5
[ 1.340527] iommu: Adding device 0000:02:00.0 to group 5
[ 1.340538] iommu: Adding device 0000:03:00.0 to group 5
[ 1.340549] iommu: Adding device 0000:04:00.0 to group 5
# lspci -tv
-[0000:00]-+-00.0 Intel Corporation Atom/Celeron/Pentium Processor
N4200/N3350/E3900 Series Host Bridge
+-02.0 Intel Corporation Device 5a85
+-0e.0 Intel Corporation Atom/Celeron/Pentium Processor
N4200/N3350/E3900 Series Audio Cluster
+-0f.0 Intel Corporation Atom/Celeron/Pentium Processor
N4200/N3350/E3900 Series Trusted Execution Engine
+-12.0 Intel Corporation Atom/Celeron/Pentium Processor
N4200/N3350/E3900 Series SATA AHCI Controller
+-13.0-[01]----00.0 Realtek Semiconductor Co., Ltd.
RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
+-13.1-[02]----00.0 Qualcomm Atheros QCA6174 802.11ac Wireless
Network Adapter
+-13.2-[03]----00.0 ASMedia Technology Inc. ASM1062 Serial ATA
Controller
+-13.3-[04]----00.0 Realtek Semiconductor Co., Ltd.
RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
+-15.0 Intel Corporation Atom/Celeron/Pentium Processor
N4200/N3350/E3900 Series USB xHCI
+-1f.0 Intel Corporation Atom/Celeron/Pentium Processor
N4200/N3350/E3900 Series Low Pin Count Interface
\-1f.1 Intel Corporation Atom/Celeron/Pentium Processor
N4200/N3350/E3900 Series SMBus Controller
# lspci -vvnnns 00:13.3
00:13.3 PCI bridge [0604]: Intel Corporation Atom/Celeron/Pentium Processor
N4200/N3350/E3900 Series PCI Express Port A #4 [8086:5adb] (rev fb)
(prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin D routed to IRQ 125
Bus: primary=00, secondary=04, subordinate=04, sec-latency=0
I/O behind bridge: 0000c000-0000cfff
Memory behind bridge: 91300000-913fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort+ <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag- RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 256 bytes, MaxReadReq 128 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq- AuxPwr+
TransPend-
LnkCap: Port #6, Speed 5GT/s, Width x1, ASPM not supported,
Exit Latency L0s <1us, L1 <4us
ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+
DLActive+ BWMgmt+ ABWMgmt-
SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug-
Surprise-
Slot #5, PowerLimit 10.000W; Interlock- NoCompl+
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt-
HPIrq- LinkChg-
Control: AttnInd Unknown, PwrInd Unknown, Power-
Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+
Interlock-
Changed: MRL- PresDet- LinkState+
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+
CRSVisible-
RootCap: CRSVisible-
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Range ABC, TimeoutDis+, LTR+,
OBFF Not Supported ARIFwd+
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
LTR+, OBFF Disabled ARIFwd-
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance-
SpeedDis-
Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB,
EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-,
LinkEqualizationRequest-
Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
Address: fee002d8 Data: 0000
Capabilities: [90] Subsystem: ASRock Incorporation
Atom/Celeron/Pentium Processor N4200/N3350/E3900 Series PCI Express Port A
[1849:5adb]
Capabilities: [a0] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [100 v0] #00
Capabilities: [140 v1] Access Control Services
ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+
UpstreamFwd- EgressCtrl- DirectTrans-
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir-
UpstreamFwd- EgressCtrl- DirectTrans-
Capabilities: [150 v0] #00
Capabilities: [200 v1] L1 PM Substates
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+
L1_PM_Substates+
PortCommonModeRestoreTime=40us
PortTPowerOnTime=10us
L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
T_CommonMode=0us LTR1.2_Threshold=0ns
L1SubCtl2: T_PwrOn=10us
Kernel driver in use: pcieport
Kernel modules: shpchp
What I already did and tried:
– Updated the BIOS and turned on intel virtualization (VT-x, VT-d)
– updated the grub with intel_iommu=on iommu=pt
pcie_acs_override=downstream (tried downstream,multifunction as well)
– loaded vfio, vfio_iommu_type1, vfio_pci, vfio_virqfd kernel modules at
boot time (/etc/modules)
– write: options vfio_iommu_type1 allow_unsafe_interrupts=1 to
/etc/modprobe.d/iommu_unsafe_interrupts.conf
– write: options vfio-pci ids=[8086:5ad9],[8086:5ada] to
/etc/modprobe.d/vfio.conf
Do you have any idea what could be the problem?
Thank you
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20190223/b836b281/attachment.htm>
More information about the vfio-users
mailing list