[vfio-users] VFIO-PCI with AARCH64 QEMU

Haynal, Steve Steve_Haynal at mentor.com
Thu Oct 27 00:24:46 UTC 2016

Hi All,

I was able to enable both memory regions but my test program did not work on aarch64 as it does on x86. The driver is an UIO driver and it fails when it can't find resource0 in /sys/bus/pci/devices/0000:00:09.0. In the x86 guest, I see resource0 and resource1 in that directory. In the aarch64 guest, there is no resourceN. Is this related? 

Any ideas on how I can have resource0 and resource1 populated? 

The memory regions still show up as disabled and I must enable them with setpci. The pci-related portion of the kernel log is below as well as lspci output. I updated to kernel 4.8.4-040804. I see the same behavior with 4.4.0 or 4.8.4.

Best Regards,

Steve Haynal

[    5.473692] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[    5.473848] pciehp: PCI Express Hot Plug Controller Driver version: 0.4
[    5.475878] OF: PCI: host bridge /pcie at 10000000 ranges:
[    5.476320] OF: PCI:    IO 0x3eff0000..0x3effffff -> 0x00000000
[    5.476616] OF: PCI:   MEM 0x10000000..0x3efeffff -> 0x10000000
[    5.476678] OF: PCI:   MEM 0x8000000000..0xffffffffff -> 0x8000000000
[    5.477429] pci-host-generic 3f000000.pcie: ECAM at [mem 0x3f000000-0x3fffffff] for [bus 00-0f]
[    5.479081] pci-host-generic 3f000000.pcie: PCI host bridge to bus 0000:00
[    5.479354] pci_bus 0000:00: root bus resource [bus 00-0f]
[    5.479460] pci_bus 0000:00: root bus resource [io  0x0000-0xffff]
[    5.479496] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]
[    5.479524] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]
[    5.480416] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000
[    5.483773] pci 0000:00:09.0: [10ee:7022] type 00 class 0x058000
[    5.484100] pci 0000:00:09.0: reg 0x10: [mem 0x10800000-0x10800fff]
[    5.484163] pci 0000:00:09.0: reg 0x14: [mem 0x10000000-0x107fffff]
[    5.488027] pci 0000:00:09.0: BAR 1: assigned [mem 0x10000000-0x107fffff]
[    5.488274] pci 0000:00:09.0: BAR 0: assigned [mem 0x10800000-0x10800fff]

lspci -vvv
00:00.0 Host bridge: Red Hat, Inc. Device 0008
	Subsystem: Red Hat, Inc Device 1100
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

00:09.0 Memory controller: Xilinx Corporation Device 7022
	Subsystem: Xilinx Corporation Device 0007
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin A routed to IRQ 47
	Region 0: Memory at 10800000 (32-bit, non-prefetchable) [disabled] [size=4K]
	Region 1: Memory at 10000000 (32-bit, non-prefetchable) [disabled] [size=8M]
	Capabilities: [80] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [90] MSI: Enable- Count=1/1 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [c0] Express (v2) Root Complex Integrated Endpoint, MSI 00
		DevCap:	MaxPayload 512 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported+
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 256 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		DevCap2: Completion Timeout: Range B, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis-, LTR-, OBFF Disabled
	Capabilities: [100 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt+ UnxCmplt+ RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP+ FCP+ CmpltTO+ CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-

-----Original Message-----
From: Haynal, Steve 
Sent: Tuesday, October 25, 2016 5:16 PM
To: 'Laszlo Ersek'; Ard Biesheuvel
Cc: Alex Williamson; vfio-users at redhat.com; Eric Auger
Subject: RE: [vfio-users] VFIO-PCI with AARCH64 QEMU

Hi All,

I can enable the memory region with the "setpci -s 00:09.0 COMMAND=2:2" command. For proof of concept tests, I can get by with a shared memory size of 8MB, which should fit. I can also switch to 64-bit BARs. Both of these changes require resynthesizing the FPGA design overnight and may cause other problems, so I will report back if it works tomorrow.

I am using the stock default kernel in the current Xenial aarch64 cloud image from Ubuntu (4.4.0-45). I will build a newer kernel.

I also prefer the enumeration and standardization of a PCI device over a platform device, but some of our customers want the virtual environment to more closely match their final hardware target environment. I will take a look at ivshmem. 

Thanks again to all for the help.
Best Regards,

Steve Haynal

More information about the vfio-users mailing list