[vfio-users] VFIO-PCI with AARCH64 QEMU

[Ard Biesheuvel]

On 25 October 2016 at 21:38, Haynal, Steve <Steve_Haynal at mentor.com> wrote:
> Hi All,
>
> Thanks for the help. I've started using explicit pflash drives instead of -bios. The firmware I was using was 15.12 from https://releases.linaro.org/components/kernel/uefi-linaro/15.12/release/qemu64/QEMU_EFI.fd. This was not producing any interesting debug output, so I built my own from git following these instructions https://wiki.linaro.org/LEG/UEFIforQEMU . This produces the output shown below. Once booted, the lspci output still looks the same as before. If I add acpi=force during boot or compile with -D PURE_ACPI_BOOT_ENABLE, the boot always hangs at the line " EFI stub: Exiting boot services and installing virtual address map..."  Boot completes without these options. Any ideas on why the memory regions show up as disabled in lspci, and why the large 512MB region is ignored?
>
> The 512MB memory region is quite a bit to reserve. We have Google's BigE hardware IP (see https://www.webmproject.org/hardware/vp9/bige/) running on an FPGA. This IP shares memory with the host and currently Google's driver allocates memory from this 512MB region when it must be shared between the application and IP on the FPGA. We want to test this IP on a virtual aarch64 platform and hence the device pass through and interest in vfio. Eventually, we'd like these passed through memory regions to appear as platform devices. Is it possible/recommended to hack the vfio infrastructure such that a PCI device on the host side appears as a platform device in an aarch64 Qemu machine? We've done something similar with virtual device drivers. Should we stick with virtual device drivers?
>

While informative, the way the firmware handles the PCI resource
allocation is not highly relevant, given that you're not booting from
the device, and on arm64, the kernel will reallocate all PCI resources
anyway.

The relevant bit from the kernel log is

[   62.992533] pci 0000:00:09.0: BAR 1: no space for [mem size 0x20000000]
[   62.992669] pci 0000:00:09.0: BAR 1: failed to assign [mem size 0x20000000]

The 32-bit window for MMIO BAR allocation is simply not large enough
to allocate both BARs in a way that adheres to all range and alignment
requirements. It looks as if on arm64, the BARs are not sorted by
size/alignment, but are simply allocated in order, which means
naturally aligning the 512 MB wastes ~512 MB of 32-bit MMIO space,
leaving insufficient space. On x86, the BARs are apparently allocated
in a saner order.