[vfio-users] CPU stall on dynamic rebinding with nvidia proprietary drivers

[Torbjorn Jansson]

On 2018-07-26 12:56, Alex John wrote:
> Hello!
> 
> I was experimenting with dynamically rebinding my GPU (nvidia <-> vfio-pci) and
> it works exactly two times and crashes on the third time. More details and
> kernel logs as follows:
> 
> I boot the system using the 1080 GTX as the boot GPU, X starts fine, everything
> is useable. When I need to boot up one of VMs I
> 
>    * first kill X server, and wait for it to completely shut down
>    * unbind the device from the nvidia driver
>    * bind it to vfio-pci
>    * do the same for the HD audio device
>    * unbind the framebuffer device by doing
>        echo "efi-framebuffer.0" > \
>          /sys/bus/platform/drivers/efi-framebuffer/unbind
>    * restart X with a different configuration file that starts it on the intel
>      iGPU (i965)
> 
> This works fine. I get a vtconsole that is modesetted by the intel driver while
> I'm working in the iGPU. Once done, I kill X again, rebind the card to the
> nvidia driver and start X on it. All good uptil this point. However, I've lost
> virtual console at this point and if I try to drop to it using Ctrl+Alt+F1 etc
> my CPU stalls. The relevant snipped portion of the log can be found below.
> 
> The full log is also at: https://bpaste.net/show/0f80d62444df
> 
> If anyone has encountered this before any input would be appreciated. Thank you!
> 
> Alex

i think dynamic rebinding of nvidia gfx cards is a really bad idea and i don't 
think you can expect it to work.

in my experience for things to work well with assigning a physical gfx card to 
a vm nothing must touch the card other than vfio-pci.
if you let nvidia driver bind to it then you will have problems.