[vfio-users] VFIO-PCI with AARCH64 QEMU

[Haynal, Steve]

Hi Ard,

Yes, I noted that message in my first e-mail too. I may be able to reduce the size of the region to something that will work on aarch64. But even for another device that has a single small (16kB) memory region, that region shows up as disabled in lspci on aarch64 but not so on x86. There are no drivers in either case. Any ideas on why it shows up as disabled on aarch64?

Best Regards,

Steve Haynal


-----Original Message-----
From: Ard Biesheuvel [mailto:ard.biesheuvel at linaro.org] 
Sent: Tuesday, October 25, 2016 2:10 PM
To: Haynal, Steve
Cc: Laszlo Ersek; Alex Williamson; vfio-users at redhat.com; Eric Auger
Subject: Re: [vfio-users] VFIO-PCI with AARCH64 QEMU

On 25 October 2016 at 21:38, Haynal, Steve <Steve_Haynal at mentor.com> wrote:
> Hi All,
>
> Thanks for the help. I've started using explicit pflash drives instead of -bios. The firmware I was using was 15.12 from https://releases.linaro.org/components/kernel/uefi-linaro/15.12/release/qemu64/QEMU_EFI.fd. This was not producing any interesting debug output, so I built my own from git following these instructions https://wiki.linaro.org/LEG/UEFIforQEMU . This produces the output shown below. Once booted, the lspci output still looks the same as before. If I add acpi=force during boot or compile with -D PURE_ACPI_BOOT_ENABLE, the boot always hangs at the line " EFI stub: Exiting boot services and installing virtual address map..."  Boot completes without these options. Any ideas on why the memory regions show up as disabled in lspci, and why the large 512MB region is ignored?
>
> The 512MB memory region is quite a bit to reserve. We have Google's BigE hardware IP (see https://www.webmproject.org/hardware/vp9/bige/) running on an FPGA. This IP shares memory with the host and currently Google's driver allocates memory from this 512MB region when it must be shared between the application and IP on the FPGA. We want to test this IP on a virtual aarch64 platform and hence the device pass through and interest in vfio. Eventually, we'd like these passed through memory regions to appear as platform devices. Is it possible/recommended to hack the vfio infrastructure such that a PCI device on the host side appears as a platform device in an aarch64 Qemu machine? We've done something similar with virtual device drivers. Should we stick with virtual device drivers?
>

While informative, the way the firmware handles the PCI resource allocation is not highly relevant, given that you're not booting from the device, and on arm64, the kernel will reallocate all PCI resources anyway.

The relevant bit from the kernel log is

[   62.992533] pci 0000:00:09.0: BAR 1: no space for [mem size 0x20000000]
[   62.992669] pci 0000:00:09.0: BAR 1: failed to assign [mem size 0x20000000]

The 32-bit window for MMIO BAR allocation is simply not large enough to allocate both BARs in a way that adheres to all range and alignment requirements. It looks as if on arm64, the BARs are not sorted by size/alignment, but are simply allocated in order, which means naturally aligning the 512 MB wastes ~512 MB of 32-bit MMIO space, leaving insufficient space. On x86, the BARs are apparently allocated in a saner order.