[vfio-users] [OVMF+VIFO+QEMU] Large VM RAM allocation = Long Boot Times

[Alex Williamson]

On Wed, 14 Nov 2018 21:59:08 -0800
A Z <adam at zegelin.com> wrote:

>  This is an issue that involves a combination of different software
> packages, so my apologies in advance if this is the wrong list to post on.
> 
> I'm experiencing terrible boot times when I assign a large amount of RAM to
> a VM when used in combination with VIFO/PCI-passthrough.
> 
> On a VM with a Nvidia GTX 970 + USB controller and 24GiB of RAM assigned,
> the time to the TianoCore splash screen is ~5 minutes. It's then ~30
> seconds before Windows 10 begins to boot (spinning dots). During this time,
> the QEMU CPU core threads are 100% busy.
> 
> According to `perf`, the QMU CPU core threads are spending most of their
> time waiting on a spinlock over kvm->mmu_lock that's created by
> kvm_zap_gfn_range.
> 
> I'm fairly certain that ~1 year ago (if not longer) the same configuration
> didn't take this long to boot.


If you can confirm that your system uses a NUMA memory configuration
and that 24G exceeds the node local memory, then this is a known
regression in the Linux kernel mm.  AIUI it's being worked upstream,
workarounds are to either use hugepages or locked memory for the VM.
Thanks,

Alex