[vfio-users] vfio pass through only works once after reboot - more tests

[Alex Williamson]

On Thu, 2015-09-17 at 23:34 +0200, Karsten Elfenbein wrote:
> 06:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce
> GTX 970] (rev a1) (prog-if 00 [VGA controller])
>         Subsystem: Device 196e:1131
>         Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx-
>         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>         Latency: 0
>         Interrupt: pin A routed to IRQ 30
>         Region 0: Memory at f9000000 (32-bit, non-prefetchable) [size=16M]
>         Region 1: Memory at d0000000 (64-bit, prefetchable) [size=256M]
>         Region 3: Memory at e0000000 (64-bit, prefetchable) [size=32M]
>         Region 5: I/O ports at d000 [size=128]
>         [virtual] Expansion ROM at fa000000 [disabled] [size=512K]
>         Capabilities: [60] Power Management version 3
>                 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
> PME(D0-,D1-,D2-,D3hot-,D3cold-)
>                 Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
>         Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+
>                 Address: 00000000fee00398  Data: 0000
>         Capabilities: [78] Express (v2) Legacy Endpoint, MSI 00
>                 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
> unlimited, L1 <64us
>                         ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
>                 DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
> Unsupported-
>                         RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop-
>                         MaxPayload 256 bytes, MaxReadReq 512 bytes
>                 DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq-
> AuxPwr- TransPend-
>                 LnkCap: Port #0, Speed 5GT/s, Width x16, ASPM L0s L1,
> Exit Latency L0s <512ns, L1 <4us
>                         ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
>                 LnkCtl: ASPM L1 Enabled; RCB 64 bytes Disabled- CommClk+
>                         ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
>                 LnkSta: Speed 5GT/s, Width x16, TrErr- Train- SlotClk+
> DLActive- BWMgmt- ABWMgmt-
>                 DevCap2: Completion Timeout: Range AB, TimeoutDis+,
> LTR+, OBFF Via message
>                 DevCtl2: Completion Timeout: 50us to 50ms,
> TimeoutDis-, LTR-, OBFF Disabled
>                 LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
>                          Transmit Margin: Normal Operating Range,
> EnterModifiedCompliance- ComplianceSOS-
>                          Compliance De-emphasis: -6dB
>                 LnkSta2: Current De-emphasis Level: -6dB,
> EqualizationComplete-, EqualizationPhase1-
>                          EqualizationPhase2-, EqualizationPhase3-,
> LinkEqualizationRequest-


Something very strange here, where are your extended PCIe capabilities?
My GTX750 lists:

	Capabilities: [100] Virtual Channel
	Capabilities: [250] Latency Tolerance Reporting
	Capabilities: [258] L1 PM Substates
	Capabilities: [128] Power Budgeting <?>
	Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
	Capabilities: [900] #19

Nothing in your system has extended capabilities.  That sort of explains
the -EFAULT, though maybe it implies a bug elsewhere.  Can you pastebin
your full dmesg?

>         Kernel driver in use: nvidia
>         Kernel modules: nvidia

Hmm, keep in mind to try w/o nvidia on the host if we don't make
progress.