<div dir="ltr"><div><div>Hey everyone,<br><br></div>I use Proxmox to manage my VMs, and one of the VMs requires a passed-through HBA. Up until today I've been using Proxmox 3.4 with kernel 2.6.32, which did not use VFIO for passthrough. This set-up worked fine using an IBM M1015 (SAS2008) flashed to 9211-8i IT mode, so the card isn't faulty and is possible to be passed through. Today I updated to the latest version of Proxmox where passthrough is always done via VFIO as far as I can tell.<br><br></div>However, with VFIO it is not possible to pass through this card. A dummy command for pass-through returns:<br><br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"># qemu-system-x86_64 -enable-kvm -m 10G -net none -monitor stdio -serial none -parallel none -vnc :1 -device vfio-pci,host=1:00.0 -S<br>QEMU 2.9.1 monitor - type 'help' for more information<br>(qemu) c<br>(qemu) qemu-system-x86_64: VFIO_MAP_DMA: -14<br>qemu-system-x86_64: vfio_dma_map(0x7f548f0a1fc0, 0xfebd0000, 0x2000, 0x7f54a909d000) = -14 (Bad address)<br>qemu: hardware error: vfio: DMA mapping failed, unable to continue<br>CPU #0:<br>EAX=00000103 EBX=80001804 ECX=00000103 EDX=00000cfc<br>ESI=00001800 EDI=00000000 EBP=bffab6b0 ESP=00006d34<br>EIP=000eda90 EFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0<br>ES =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]<br>CS =0008 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA]<br>SS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]<br>DS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]<br>FS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]<br>GS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]<br>LDT=0000 00000000 0000ffff 00008200 DPL=0 LDT<br>TR =0000 00000000 0000ffff 00008b00 DPL=0 TSS32-busy<br>GDT=     000f7070 00000037<br>IDT=     000f70ae 00000000<br>CR0=00000011 CR2=00000000 CR3=00000000 CR4=00000000<br>DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 <br>DR6=00000000ffff0ff0 DR7=0000000000000400<br>EFER=0000000000000000<br>FCW=037f FSW=0000 [ST=0] FTW=00 MXCSR=00001f80<br>FPR0=0000000000000000 0000 FPR1=0000000000000000 0000<br>FPR2=0000000000000000 0000 FPR3=0000000000000000 0000<br>FPR4=0000000000000000 0000 FPR5=0000000000000000 0000<br>FPR6=0000000000000000 0000 FPR7=0000000000000000 0000<br>XMM00=00000000000000000000000000000000 XMM01=00000000000000000000000000000000<br>XMM02=00000000000000000000000000000000 XMM03=00000000000000000000000000000000<br>XMM04=00000000000000000000000000000000 XMM05=00000000000000000000000000000000<br>XMM06=00000000000000000000000000000000 XMM07=00000000000000000000000000000000<br>Aborted<br></blockquote><div><br></div><div>This has been previously reported on this mailing list (<a href="https://www.redhat.com/archives/vfio-users/2016-May/msg00054.html">https://www.redhat.com/archives/vfio-users/2016-May/msg00054.html</a>), but no solution was found for the poster.</div><div><br></div><div>It seems that on Fedora this passthrough works fine (<a href="https://www.redhat.com/archives/vfio-users/2016-May/msg00246.html">https://www.redhat.com/archives/vfio-users/2016-May/msg00246.html</a>), but in Ubuntu and Proxmox (also Debian-based), it doesn't. Other cards, such as the 9207-8i (SAS2308 chip) also seem to work (<a href="https://forum.proxmox.com/threads/promxox4-4-qm-status-stopped-for-passthrough-of-ibm1015.32841/#post-169018">https://forum.proxmox.com/threads/promxox4-4-qm-status-stopped-for-passthrough-of-ibm1015.32841/#post-169018</a>).</div><div><br></div><div>Any ideas on how to get this card working? I'd be grateful for any input, and I'm not alone in this combination of hardware.</div><div><br></div><div><br></div><div>For completeness:</div><div style="margin-left:40px">root@pve:~# uname -a<br>Linux pve 4.13.4-1-pve #1 SMP PVE 4.13.4-26 (Mon, 6 Nov 2017 11:23:55 +0100) x86_64 GNU/Linux</div><div style="margin-left:40px"><br></div>and<br><br><div style="margin-left:40px">01:00.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] [1000:0072] (rev 03)<br>        Subsystem: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] [1000:3020]<br>        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+<br>        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-<br>        Interrupt: pin A routed to IRQ 16<br>        Region 0: I/O ports at e000 [disabled] [size=256]<br>        Region 1: Memory at df6c0000 (64-bit, non-prefetchable) [disabled] [size=16K]<br>        Region 3: Memory at df280000 (64-bit, non-prefetchable) [disabled] [size=256K]<br>        Expansion ROM at df200000 [disabled] [size=512K]<br>        Capabilities: [50] Power Management version 3<br>                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)<br>                Status: D3 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-<br>        Capabilities: [68] Express (v2) Endpoint, MSI 00<br>                DevCap: MaxPayload 4096 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us<br>                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W<br>                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-<br>                        RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-<br>                        MaxPayload 128 bytes, MaxReadReq 512 bytes<br>                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-<br>                LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s, Exit Latency L0s <64ns, L1 <1us<br>                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-<br>                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+<br>                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-<br>                LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-<br>                DevCap2: Completion Timeout: Range BC, TimeoutDis+, LTR-, OBFF Not Supported<br>                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled<br>                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-<br>                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-<br>                         Compliance De-emphasis: -6dB<br>                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-<br>                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-<br>        Capabilities: [d0] Vital Product Data<br>pcilib: sysfs_read_vpd: read failed: Input/output error<br>                Not readable<br>        Capabilities: [a8] MSI: Enable- Count=1/1 Maskable- 64bit+<br>                Address: 0000000000000000  Data: 0000<br>        Capabilities: [c0] MSI-X: Enable- Count=15 Masked-<br>                Vector table: BAR=1 offset=00002000<br>                PBA: BAR=1 offset=00003800<br>        Capabilities: [100 v1] Advanced Error Reporting<br>                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-<br>                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-<br>                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-<br>                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-<br>                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+<br>                AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-<br>        Capabilities: [138 v1] Power Budgeting <?><br>        Capabilities: [150 v1] Single Root I/O Virtualization (SR-IOV)<br>                IOVCap: Migration-, Interrupt Message Number: 000<br>                IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy-<br>                IOVSta: Migration-<br>                Initial VFs: 16, Total VFs: 16, Number of VFs: 0, Function Dependency Link: 00<br>                VF offset: 1, stride: 1, Device ID: 0072<br>                Supported Page Size: 00000553, System Page Size: 00000001<br>                Region 0: Memory at 00000000df6c4000 (64-bit, non-prefetchable)<br>                Region 2: Memory at 00000000df2c0000 (64-bit, non-prefetchable)<br>                VF Migration: offset: 00000000, BIR: 0<br>        Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI)<br>                ARICap: MFVC- ACS-, Next Function: 0<br>                ARICtl: MFVC- ACS-, Function Group: 0<br>        Kernel driver in use: vfio-pci<br>        Kernel modules: mpt3sas<br></div><br></div>