[vfio-users] two nvidia cards, one for the host with nvidia module

[ddio at jr0.de]

Hi,

I have two Nvidia Cards ( GTX750 Ti and GTX960 ). I want to passthrough 
the GTX960 into a vm with windows, the other one should be used for the 
Host with the propietary kernel module of Nvidia.

My settings:

Boot options:

GRUB_CMDLINE_LINUX="intel_iommu=on i915.enable_hd_vgaarb=1 
init=/usr/lib/systemd/systemd"

When I prepare the 2 pcie devices with: (01:00.0 being the GTX750 Ti and 
02:00.0 the GTX960)

echo '0000:02:00.0' | sudo tee 
/sys/bus/pci/devices/0000:02:00.0/driver/unbind
echo '0000:02:00.1' | sudo tee 
/sys/bus/pci/devices/0000:02:00.1/driver/unbind    #(hdmi audio device)

echo 10de 1401 | tee /sys/bus/pci/drivers/vfio-pci/new_id
echo 10de 0fba | tee /sys/bus/pci/drivers/vfio-pci/new_id

I can see that for kernel driver in use it then is vfio-pci for both 
devices, but for the graphic device there is still kernel module in use: 
nvidia.

I guess that is my problem, because the vm freezes instantly ( the 
cursor in the monitor window blinks two times and then stops ), and 
there is no output on the monitor connected to the GTX 960.

I need the nvidia module for the GTX750 Ti Device.

Is that my problem, or sth different? Is there a way to prevent the 
nvidia module to register for both cards?

Here is my qemu command:

qemu-system-x86_64 \
   -enable-kvm \
   -cpu host,kvm=off \
   -soundhw hda \
   -M q35,accel=kvm \
   -bios /usr/share/qemu/bios.bin \
   -smp 8,sockets=1,cores=8,threads=1 \
   -drive file=w7.qcow2,if=virtio \
   -m 2048 \
   -vga none \
   -net bridge \
   -net nic,model=virtio \
   -device vfio-pci,host=02:00.0,rombar=1,x-vga=on \
   -device vfio-pci,host=02:00.1

Thanks for help!