[vfio-users] q35, pcie 1x PCH, IOMMU confusion and freezing-host with no error log

Wed Nov 8 03:42:05 UTC 2017

first: I use proxmox 5 (pvetest)

good evening, I'm succesfully passing through a nvidia board to my q35
guest but I'm getting host-freezes after starting the vm with a hostpci1:
04:00.0

03:00.0 PCI bridge: Texas Instruments XIO2000(A)/XIO2200A PCI
Express-to-PCI Bridge (rev 03)

04:00.0 FireWire (IEEE 1394): Texas Instruments XIO2200A IEEE-1394a-2000
Controller (PHY/Link) (rev 01)

I've already set the corresponding pci-stub arguments to the kernel and
checked all that is ok.

I managed to overcome errors on 03:00 not biding when trying to pass the
04:00 device but as soon as the vm starts, the host gets completely frozen.
I have to stop my lvm group (running over a raid0) when I do tests because
the raid breaks when the host freezes, there is no log/error produced when
the crash happens.

I've read that a IOMMU group shared by a device and the pcie bridge
produces vfio to try bind the bridge to the vm and that is not possible,
but there is also a PCH device produced as a workaround by the kernel,
which I don't know how to pass to the vm, as tried in the pve-q35.cfg

lspci -tv

+-1c.4-[03-04]----00.0-[04]----00.0  Texas Instruments XIO2200A
IEEE-1394a-2000 Controller (PHY/Link)

lspci -tnv

+-1c.4-[03-04]----00.0-[04]----00.0  104c:8235

lspci -vvvv -s 04:00

04:00.0 FireWire (IEEE 1394): Texas Instruments XIO2200A IEEE-1394a-2000
Controller (PHY/Link) (rev 01) (prog-if 10 [OHCI])

Subsystem: Texas Instruments XIO2200A IEEE-1394a-2000 Controller (PHY/Link)

Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx-

Status: Cap+ 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort-
<MAbort- >SERR- <PERR- INTx-

Interrupt: pin A routed to IRQ 11

Region 0: Memory at f7904000 (32-bit, non-prefetchable) [disabled] [size=2K]

Region 1: Memory at f7900000 (32-bit, non-prefetchable) [disabled]
[size=16K]

Capabilities: [44] Power Management version 2

Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-)

Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-

Kernel driver in use: pci-stub

Kernel modules: firewire_ohci

lspci -vvvv -s 03:00

03:00.0 PCI bridge: Texas Instruments XIO2000(A)/XIO2200A PCI
Express-to-PCI Bridge (rev 03) (prog-if 00 [Normal decode])

Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx-

Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
<MAbort- >SERR- <PERR- INTx-

Latency: 0, Cache Line Size: 64 bytes

Bus: primary=03, secondary=04, subordinate=04, sec-latency=32

I/O behind bridge: 0000f000-00000fff

Memory behind bridge: f7900000-f79fffff

Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff

Secondary status: 66MHz+ FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort-
<MAbort+ <SERR- <PERR-

BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-

PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-

Capabilities: [50] Power Management version 2

Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)

Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-

Bridge: PM- B3+

Capabilities: [60] MSI: Enable- Count=1/16 Maskable- 64bit+

Address: 0000000000000000  Data: 0000

Capabilities: [80] Subsystem: Device 0000:0000

Capabilities: [90] Express (v1) PCI-Express to PCI/PCI-X Bridge, MSI 00

DevCap: MaxPayload 512 bytes, PhantFunc 0

ExtTag- AttnBtn- AttnInd- PwrInd- RBE- SlotPowerLimit 10.000W

DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-

RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ BrConfRtry-

MaxPayload 128 bytes, MaxReadReq 512 bytes

DevSta: CorrErr- UncorrErr+ FatalErr- UnsuppReq- AuxPwr+ TransPend-

LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s
<64ns, L1 <1us

ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-

LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+

ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-

LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt-
ABWMgmt-

Capabilities: [100 v1] Advanced Error Reporting

UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
ECRC- UnsupReq- ACSViol-

UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
ECRC- UnsupReq- ACSViol-

UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+
ECRC- UnsupReq- ACSViol-

CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-

CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-

AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-

Kernel modules: shpchp

lspci -vvvv -s 1c.4

00:1c.4 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family
PCI Express Root Port #5 (rev d5) (prog-if 00 [Normal decode])

Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx+

Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
<MAbort- >SERR- <PERR- INTx-

Latency: 0, Cache Line Size: 64 bytes

Interrupt: pin A routed to IRQ 28

Bus: primary=00, secondary=03, subordinate=04, sec-latency=0

I/O behind bridge: 0000f000-00000fff

Memory behind bridge: f7900000-f79fffff

Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff

Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
<MAbort+ <SERR- <PERR-

BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-

PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-

Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00

DevCap: MaxPayload 128 bytes, PhantFunc 0

ExtTag- RBE+

DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-

RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-

MaxPayload 128 bytes, MaxReadReq 128 bytes

DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-

LnkCap: Port #5, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s
<512ns, L1 <16us

ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp-

LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+

ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-

LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt+
ABWMgmt-

SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-

Slot #4, PowerLimit 10.000W; Interlock- NoCompl+

SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-

Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-

SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-

Changed: MRL- PresDet- LinkState-

RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-

RootCap: CRSVisible-

RootSta: PME ReqID 0000, PMEStatus- PMEPending-

DevCap2: Completion Timeout: Range ABC, TimeoutDis+, LTR+, OBFF Not
Supported ARIFwd-

DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled
ARIFwd-

LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-

Transmit Margin: Normal Operating Range, EnterModifiedCompliance-
ComplianceSOS-

Compliance De-emphasis: -6dB

LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-,
EqualizationPhase1-

EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-

Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-

Address: fee00258  Data: 0000

Capabilities: [90] Subsystem: Elitegroup Computer Systems 8 Series/C220
Series Chipset Family PCI Express Root Port

Capabilities: [a0] Power Management version 3

Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)

Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-

Kernel driver in use: pcieport

Kernel modules: shpchp

pve-q35.cfg attempt to assign the pch device tree

[device "pcie.2"]

  driver = "pxb-pcie"

  multifunction = "on"

  addr = "1c.4"

  bus_nr = "0x08"

[device "pci.21"]

  driver = "ioh3420"

  multifunction = "on"

  bus = "pcie.2"

  addr = "0.3"

  slot = "0x04"

  port = "5"

  chassis = "5"

I've read the docs about the different approaches for virtual pcie ports,
upstream and downstream switches and root ports, but the guest device
manager reports the virtual ports only, still not getting the firewire
board visible in the guest.

I don't understand completely how can the virtual pcie ports be mapped to
that 1c.4 [03-04] tree so I thought this would be the best place to get
some light on the subject.

the good part is that the host doesn't freeze when using the pve-q35.cfg
approach to assign the device to the guest, but even if I get some
close-numbered dev-ids assigned to the virtual pcie ports, they still don't
relate to the physical device.

this is the line about pch in dmesg:
Kernel: [    0.080766] pci 0000:00:1c.4: Intel PCH root port ACS workaround
enabled

here is the dmesg log for the vfio-pci and pci-stub modules

[    1.704898] vfio_pci: add [10de:1287[ffff:ffff]] class 0x000000/00000000

[    1.704901] vfio_pci: add [10de:0e0f[ffff:ffff]] class 0x000000/00000000

[    1.704902] vfio_pci: add [1019:7e44[ffff:ffff]] class 0x000000/00000000

[    1.704904] vfio_pci: add [104c:8235[ffff:ffff]] class 0x000000/00000000

[    1.701664] pci-stub: add 10DE:1287 sub=FFFFFFFF:FFFFFFFF
cls=00000000/00000000

[    1.701674] pci-stub 0000:01:00.0: claimed by stub

[    1.701680] pci-stub: add 10DE:0E0F sub=FFFFFFFF:FFFFFFFF
cls=00000000/00000000

[    1.701684] pci-stub 0000:01:00.1: claimed by stub

[    1.701686] pci-stub: add 1019:7E44 sub=FFFFFFFF:FFFFFFFF
cls=00000000/00000000

[    1.701688] pci-stub: add 104C:8235 sub=FFFFFFFF:FFFFFFFF
cls=00000000/00000000

[    1.701691] pci-stub 0000:04:00.0: claimed by stub
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20171108/a11346d3/attachment.htm>