[vfio-users] VFIO-PCI with AARCH64 QEMU
lersek at redhat.com
Tue Oct 25 22:01:13 UTC 2016
On 10/25/16 23:10, Ard Biesheuvel wrote:
> On 25 October 2016 at 21:38, Haynal, Steve <Steve_Haynal at mentor.com> wrote:
>> Hi All,
>> Thanks for the help. I've started using explicit pflash drives
>> instead of -bios. The firmware I was using was 15.12 from
>> This was not producing any interesting debug output, so I built my
>> own from git following these instructions
>> https://wiki.linaro.org/LEG/UEFIforQEMU . This produces the output
>> shown below. Once booted, the lspci output still looks the same as
>> before. If I add acpi=force during boot or compile with -D
>> PURE_ACPI_BOOT_ENABLE, the boot always hangs at the line " EFI
>> stub: Exiting boot services and installing virtual address map..."
>> Boot completes without these options. Any ideas on why the memory
>> regions show up as disabled in lspci, and why the large 512MB
>> region is ignored?
>> The 512MB memory region is quite a bit to reserve. We have Google's
>> BigE hardware IP (see
>> https://www.webmproject.org/hardware/vp9/bige/) running on an FPGA.
>> This IP shares memory with the host and currently Google's driver
>> allocates memory from this 512MB region when it must be shared
>> between the application and IP on the FPGA. We want to test this IP
>> on a virtual aarch64 platform and hence the device pass through and
>> interest in vfio. Eventually, we'd like these passed through memory
>> regions to appear as platform devices. Is it possible/recommended
>> to hack the vfio infrastructure such that a PCI device on the host
>> side appears as a platform device in an aarch64 Qemu machine? We've
>> done something similar with virtual device drivers. Should we stick
>> with virtual device drivers?
> While informative, the way the firmware handles the PCI resource
> allocation is not highly relevant, given that you're not booting from
> the device, and on arm64, the kernel will reallocate all PCI resources
It was me that asked for the firmware log. I had known (from you) about
the arm64 kernel reallocating PCI resources unconditionally, but I
wanted to see if the firmware encountered the same symptoms.
It did, apparently. Had it not, that would have implied a problem with
the guest kernel. (This is what I was trying to discern.)
> The relevant bit from the kernel log is
> [ 62.992533] pci 0000:00:09.0: BAR 1: no space for [mem size 0x20000000]
> [ 62.992669] pci 0000:00:09.0: BAR 1: failed to assign [mem size 0x20000000]
> The 32-bit window for MMIO BAR allocation is simply not large enough
> to allocate both BARs in a way that adheres to all range and alignment
> requirements. It looks as if on arm64, the BARs are not sorted by
I think I disagree.
While I only assume that the arm64 kernel does the same sorting+grouping
in decreasing alignment order as the x86_64 kernel does, I know for a
fact that the edk2 PCI Bus driver does the sorting+grouping regardless
of architecture. The cause is different; namely:
The 32-bit MMIO aperture exposed by "qemu-system-aarch64 -M virt" is:
[0x1000_0000, 0x3EFE_FFFF] == [ 256 MB, 1007 MB + 960 KB )
In this range, the only base address that satisfies the 512MB alignment
is 512MB itself, i.e., 0x2000_0000. And, from this base address up,
there isn't enough room left in the aperture for the 512MB size of the
BAR (there's only 495 MB + 960 KB free).
> but are simply allocated in order, which means
> naturally aligning the 512 MB wastes ~512 MB of 32-bit MMIO space,
> leaving insufficient space. On x86, the BARs are apparently allocated
> in a saner order.
I agree there isn't enough room left, but it's not due to lack of
sorting. The reason is that, given the specific boundaries of the 32-bit
MMIO aperture, the largest BAR that can be accommodated is 256MB in size.
More information about the vfio-users