[vfio-users] Power Management of vfio-redirected devices

[Eric Hattemer]

So I have this weird issue where if I run my Windows guest for days in a 
row, without allowing it to blank the screen or suspend or hibernate, it 
runs at full performance.  If I suspend it and let Linux blank the 
screen, or let Windows blank the screen, or things of that nature, it 
tends to break forever.  That is, rebooting Windows or starting a fresh 
qemu process (that doesn't resume from hibernate) doesn't fix anything.  
The symptom is that the VM functions at about 1/10th speed.  It takes 
10x as long to boot. Dragging windows across the screen stutters.  The 
sound playing almost refuses to work.  Rebooting the host (Linux) makes 
everything work again until it has a chance to enter low power mode or 
whatever.  It's probably either the sound card or more likely the Nvidia 
660 that's causing the problem, since those are both vfio-pci redirected.

I fixed it once without rebooting by messing with all of:

modprobe -r vfio-pci
service acpid restart
tuned-adm profile powersaving
tuned-adm profile vm-host
modprobe nouveau
modprobe -r nouveau
modprobe vfio-pci

and maybe some rescanning/disabling/enabling or things in /sys/bus/pci...

I'm really not sure how I did that and I haven't really been able to fix 
it without a reboot since.  The important thing to note about modprobing 
nouveau is that the sysfs node for the video card still had 
driver_override=vfio-pci.  Otherwise nouveau really does take it over 
and there doesn't seem to be a way to unload nouveau once it's attached 
to a card.

I ran MSI afterburner in the guest to look at the GPU clocks and see if 
I could get it to re-activate them to full speed.  But it might not 
really be a clocking issue, since I see about the same graphs whether 
the issue is present or not.

Maybe my hypothesis that this is GPU power-saving related is wrong, but 
everything seems to run full speed if I disable vfio and run with the 
spice graphics card.

So I'm wondering what exactly would cause this and what I should do to 
avoid/fix it.  I'm not that familiar with low-power modes and how 
they're managed.  Is this something well-understood, or am I the only 
one having this problem?

-Eric Hattemer