[vfio-users] VFIO-PCI with AARCH64 QEMU

Haynal, Steve Steve_Haynal at mentor.com
Tue Oct 25 00:29:07 UTC 2016


Hi,

I am using VFIO-PCI to pass through a PCIe endpoint on an FPGA card to virtual x86 and aarch64 QEMU instances. It works fine on x86 but I have problems with aarch64.  On aarch64, memory for both BARs on the device shows up as disabled, and one BAR is ignored. From this presentation<https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=0ahUKEwi-vvyD1_TPAhVIy2MKHZmCCt4QFggcMAA&url=https%3A%2F%2Fwww.linux-kvm.org%2Fimages%2Fb%2Fb4%2F2012-forum-VFIO.pdf&usg=AFQjCNFpT9XRMMrhxR0U7_y1tIqXNp13ww&sig2=-4egMzF5EO-DRy5BJizgvg&bvm=bv.136593572,d.cGc>, I'm under the impression that I can remap PCI devices without KVM to a virtual aarch64 machine on an x86 host. Is this possible? Below are details for my problem:

Host is running Ubuntu Xenial, 4.4 Kernel.
Both virtual machines are also Ubuntu Xenial, 4.4 Kernel, from Ubuntu Cloud Images. Except for architecture, they are pretty much identical.
Qemu is 2.7.50 from git.

Command line to start qemu for x86 machine:
qemu-system-x86_64 -enable-kvm -net nic -net user -hda disk.img -hdb my-seed.img -m 1024 -smp 2 -device vfio-pci,host=01:00.0,addr=09.0,multifunction=on -redir tcp:2223::22

Lspci -vvv for remapped device in x86 machine. Note both regions are enabled.
00:09.0 Memory controller: Xilinx Corporation Device 7022
               Subsystem: Xilinx Corporation Device 0007
               Physical Slot: 9
               Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
               Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
               Interrupt: pin A routed to IRQ 10
               Region 0: Memory at e0071000 (32-bit, non-prefetchable) [size=4K]
               Region 1: Memory at c0000000 (32-bit, non-prefetchable) [size=512M]
               Capabilities: [80] Power Management version 3
                              Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                              Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
               Capabilities: [90] MSI: Enable- Count=1/1 Maskable- 64bit+
                              Address: 0000000000000000  Data: 0000
               Capabilities: [c0] Express (v2) Endpoint, MSI 00
                              DevCap:               MaxPayload 512 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
                                             ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
                              DevCtl:  Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported+
                                             RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
                                             MaxPayload 256 bytes, MaxReadReq 512 bytes
                              DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
                              LnkCap: Port #0, Speed 2.5GT/s, Width x8, ASPM not supported, Exit Latency L0s unlimited, L1 unlimited
                                             ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                              LnkCtl:   ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                                             ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                              LnkSta:  Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                              DevCap2: Completion Timeout: Range B, TimeoutDis+, LTR-, OBFF Not Supported
                              DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis-, LTR-, OBFF Disabled
                              LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
                                             EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-


Command line to start qemu for aarch64 machine:
qemu-system-aarch64 -smp 2 -m 2048 -M virt -bios QEMU_EFI.fd -device virtio-blk-device,drive=image -drive if=none,id=image,file=disk.img -device virtio-blk-device,drive=cloud -drive if=none,id=cloud,file=cloud.img -netdev user,id=user0 -device virtio-net-device,netdev=user0 -redir tcp:2222::22 -cpu cortex-a57 -device vfio-pci,host=01:00.0,addr=09.0,multifunction=on

Lspci -vvv for remapped device in aarch64 machine. Note both regions are disabled, one is ignored.
00:09.0 Memory controller: Xilinx Corporation Device 7022
               Subsystem: Xilinx Corporation Device 0007
               Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
               Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
               Interrupt: pin A routed to IRQ 47
               Region 0: Memory at 10000000 (32-bit, non-prefetchable) [disabled] [size=4K]
               Region 1: Memory at <ignored> (32-bit, non-prefetchable) [disabled]
               Capabilities: [80] Power Management version 3
                              Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                              Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
               Capabilities: [90] MSI: Enable- Count=1/1 Maskable- 64bit+
                              Address: 0000000000000000  Data: 0000
               Capabilities: [c0] Express (v2) Root Complex Integrated Endpoint, MSI 00
                              DevCap:               MaxPayload 512 bytes, PhantFunc 0
                                             ExtTag- RBE+
                              DevCtl:  Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported+
                                             RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
                                             MaxPayload 256 bytes, MaxReadReq 512 bytes
                              DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
                              DevCap2: Completion Timeout: Range B, TimeoutDis+, LTR-, OBFF Not Supported
                              DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis-, LTR-, OBFF Disabled
               Capabilities: [100 v2] Advanced Error Reporting
                              UESta:   DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                              UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt+ UnxCmplt+ RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                              UESvrt:  DLP+ SDES+ TLP+ FCP+ CmpltTO+ CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                              CESta:   RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
                              CEMsk:  RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr+
                              AERCap:               First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-


On the aarch64 machine, when I rescan the PCI bus, I see the following in dmesg:
[  365.482929] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000
[  365.521023] pci 0000:00:09.0: [10ee:7022] type 00 class 0x058000
[  365.522971] pci 0000:00:09.0: reg 0x10: [mem 0x10000000-0x10000fff]
[  365.523213] pci 0000:00:09.0: reg 0x14: [mem 0x80000000-0x9fffffff]
[  365.539233] pci 0000:00:09.0: BAR 1: no space for [mem size 0x20000000]
[  365.539406] pci 0000:00:09.0: BAR 1: failed to assign [mem size 0x20000000]
[  365.539853] pci 0000:00:09.0: BAR 0: assigned [mem 0x10000000-0x10000fff]


Is 512MB too much for aarch64? What is the limit? I tried remapping another PCI device with only a single 16kB BAR to the aarch64 machine and that showed as disabled too. How do I enable the memory regions on aarch64? There are no drivers for these devices loaded yet on either x86 or aarch64 yet the memory shows as enable don x86.


Best Regards,

Steve Haynal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20161025/4d7f3d27/attachment.htm>


More information about the vfio-users mailing list