<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=us-ascii"> <meta name="Generator" content="Microsoft Word 14 (filtered medium)"> <style></style> </head> <body lang="EN-US" link="blue" vlink="purple"> <div class="WordSection1"> Hi,<o:p></o:p> <o:p> </o:p> I am using VFIO-PCI to pass through a PCIe endpoint on an FPGA card to virtual x86 and aarch64 QEMU instances. It works fine on x86 but I have problems with aarch64. On aarch64, memory for both BARs on the device shows up as disabled, and one BAR is ignored. From <a href="https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=0ahUKEwi-vvyD1_TPAhVIy2MKHZmCCt4QFggcMAA&url=https%3A%2F%2Fwww.linux-kvm.org%2Fimages%2Fb%2Fb4%2F2012-forum-VFIO.pdf&usg=AFQjCNFpT9XRMMrhxR0U7_y1tIqXNp13ww&sig2=-4egMzF5EO-DRy5BJizgvg&bvm=bv.136593572,d.cGc"> this presentation</a>, I’m under the impression that I can remap PCI devices without KVM to a virtual aarch64 machine on an x86 host. Is this possible? Below are details for my problem:<o:p></o:p> <o:p> </o:p> Host is running Ubuntu Xenial, 4.4 Kernel.<o:p></o:p> Both virtual machines are also Ubuntu Xenial, 4.4 Kernel, from Ubuntu Cloud Images. Except for architecture, they are pretty much identical.<o:p></o:p> Qemu is 2.7.50 from git.<o:p></o:p> <o:p> </o:p> Command line to start qemu for x86 machine:<o:p></o:p> qemu-system-x86_64 -enable-kvm -net nic -net user -hda disk.img -hdb my-seed.img -m 1024 -smp 2 -device vfio-pci,host=01:00.0,addr=09.0,multifunction=on -redir tcp:2223::22<o:p></o:p> <o:p> </o:p> Lspci –vvv for remapped device in x86 machine. Note both regions are enabled.<o:p></o:p> 00:09.0 Memory controller: Xilinx Corporation Device 7022<o:p></o:p> Subsystem: Xilinx Corporation Device 0007<o:p></o:p> Physical Slot: 9<o:p></o:p> Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-<o:p></o:p> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-<o:p></o:p> Interrupt: pin A routed to IRQ 10<o:p></o:p> Region 0: Memory at e0071000 (32-bit, non-prefetchable) [size=4K]<o:p></o:p> Region 1: Memory at c0000000 (32-bit, non-prefetchable) [size=512M]<o:p></o:p> Capabilities: [80] Power Management version 3<o:p></o:p> Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)<o:p></o:p> Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-<o:p></o:p> Capabilities: [90] MSI: Enable- Count=1/1 Maskable- 64bit+<o:p></o:p> Address: 0000000000000000 Data: 0000<o:p></o:p> Capabilities: [c0] Express (v2) Endpoint, MSI 00<o:p></o:p> DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us<o:p></o:p> ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-<o:p></o:p> DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported+<o:p></o:p> RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+<o:p></o:p> MaxPayload 256 bytes, MaxReadReq 512 bytes<o:p></o:p> DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-<o:p></o:p> LnkCap: Port #0, Speed 2.5GT/s, Width x8, ASPM not supported, Exit Latency L0s unlimited, L1 unlimited<o:p></o:p> ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+<o:p></o:p> LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+<o:p></o:p> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-<o:p></o:p> LnkSta: Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-<o:p></o:p> DevCap2: Completion Timeout: Range B, TimeoutDis+, LTR-, OBFF Not Supported<o:p></o:p> DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis-, LTR-, OBFF Disabled<o:p></o:p> LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-<o:p></o:p> EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-<o:p></o:p> <o:p> </o:p> <o:p> </o:p> Command line to start qemu for aarch64 machine:<o:p></o:p> qemu-system-aarch64 -smp 2 -m 2048 -M virt -bios QEMU_EFI.fd -device virtio-blk-device,drive=image -drive if=none,id=image,file=disk.img -device virtio-blk-device,drive=cloud -drive if=none,id=cloud,file=cloud.img -netdev user,id=user0 -device virtio-net-device,netdev=user0 -redir tcp:2222::22 -cpu cortex-a57 -device vfio-pci,host=01:00.0,addr=09.0,multifunction=on<o:p></o:p> <o:p> </o:p> Lspci –vvv for remapped device in aarch64 machine. Note both regions are disabled, one is ignored.<o:p></o:p> 00:09.0 Memory controller: Xilinx Corporation Device 7022<o:p></o:p> Subsystem: Xilinx Corporation Device 0007<o:p></o:p> Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-<o:p></o:p> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-<o:p></o:p> Interrupt: pin A routed to IRQ 47<o:p></o:p> Region 0: Memory at 10000000 (32-bit, non-prefetchable) [disabled] [size=4K]<o:p></o:p> Region 1: Memory at <ignored> (32-bit, non-prefetchable) [disabled]<o:p></o:p> Capabilities: [80] Power Management version 3<o:p></o:p> Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)<o:p></o:p> Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-<o:p></o:p> Capabilities: [90] MSI: Enable- Count=1/1 Maskable- 64bit+<o:p></o:p> Address: 0000000000000000 Data: 0000<o:p></o:p> Capabilities: [c0] Express (v2) Root Complex Integrated Endpoint, MSI 00<o:p></o:p> DevCap: MaxPayload 512 bytes, PhantFunc 0<o:p></o:p> ExtTag- RBE+<o:p></o:p> DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported+<o:p></o:p> RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+<o:p></o:p> MaxPayload 256 bytes, MaxReadReq 512 bytes<o:p></o:p> DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-<o:p></o:p> DevCap2: Completion Timeout: Range B, TimeoutDis+, LTR-, OBFF Not Supported<o:p></o:p> DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis-, LTR-, OBFF Disabled<o:p></o:p> Capabilities: [100 v2] Advanced Error Reporting<o:p></o:p> UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-<o:p></o:p> UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt+ UnxCmplt+ RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-<o:p></o:p> UESvrt: DLP+ SDES+ TLP+ FCP+ CmpltTO+ CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-<o:p></o:p> CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-<o:p></o:p> CEMsk: RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr+<o:p></o:p> AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-<o:p></o:p> <o:p> </o:p> <o:p> </o:p> On the aarch64 machine, when I rescan the PCI bus, I see the following in dmesg:<o:p></o:p> [ 365.482929] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000<o:p></o:p> [ 365.521023] pci 0000:00:09.0: [10ee:7022] type 00 class 0x058000<o:p></o:p> [ 365.522971] pci 0000:00:09.0: reg 0x10: [mem 0x10000000-0x10000fff]<o:p></o:p> [ 365.523213] pci 0000:00:09.0: reg 0x14: [mem 0x80000000-0x9fffffff]<o:p></o:p> [ 365.539233] pci 0000:00:09.0: BAR 1: no space for [mem size 0x20000000]<o:p></o:p> [ 365.539406] pci 0000:00:09.0: BAR 1: failed to assign [mem size 0x20000000]<o:p></o:p> [ 365.539853] pci 0000:00:09.0: BAR 0: assigned [mem 0x10000000-0x10000fff]<o:p></o:p> <o:p> </o:p> <o:p> </o:p> Is 512MB too much for aarch64? What is the limit? I tried remapping another PCI device with only a single 16kB BAR to the aarch64 machine and that showed as disabled too. How do I enable the memory regions on aarch64? There are no drivers for these devices loaded yet on either x86 or aarch64 yet the memory shows as enable don x86.<o:p></o:p> <o:p> </o:p> <o:p> </o:p> Best Regards,<o:p></o:p> <o:p> </o:p> Steve Haynal<o:p></o:p> </div> </body> </html>