[vfio-users] vfio PCI passthrough NVIDIA K420 GPU problems
Bronek Kozicki
brok at spamcop.net
Wed Feb 24 20:26:20 UTC 2016
On 24/02/2016 15:41, Daniel Pocock wrote:
>
>
> Hi,
>
> I'm trying to use PCI passthrough to give an NVIDIA GPU to a VM with
> qemu / KVM. I've summarized my environment below and the error I get is
> near the bottom. Any help would be appreciated.
>
> There are a few guides I've been referring to already:
> https://wiki.debian.org/VGAPassthrough
> https://www.pugetsystems.com/labs/articles/Multiheaded-NVIDIA-Gaming-using-Ubuntu-14-04-KVM-585/
> https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF
> https://bbs.archlinux.org/viewtopic.php?id=162768
> http://www.linux-kvm.org/page/VGA_device_assignment
Hi Daniel
I'm successfully passing through two Quadro M5000 (to two instances of
Windows 10), looking at
http://us.download.nvidia.com/Windows/Quadro_Certified/361.91/361.91-win10-quadro-grid-release-notes.pdf
(page 9) it is not obvious that this would work for either your K420 or
mine M5000. One gotcha - when starting Windows I do not see boot screen
at all, only Windows logon scren after nVidia drivers had loaded.
Explanation is in Alex's email sent to this list on 6th Feb 2016,
subject "No boot screen on Quadro M5000?" - basically passed through
Quadro cards are meant to be secondary only (I use them as only card,
though). However this small quirk aside, this works for me very well.
Here are details of my setup:
* distribution Arch , with packages : qemu 2.5.0 libvirt 1.3.1 (I do not
use qemu command line directly)
* my machine has 32 logical cores (2 x 8 core Xeon E5-26xx CPUs, with
HT) and 128GB ECC RAM, two Quadro M5000 cards and two USB 3.0 controller
cards; my motherboard is Supermicro X9DA7. I also dedicated half of
cores and about half of RAM to guests (I'm also running Linux virtual
machines with spice). Obviously you do not need this much hardware, just
a context to help you understand my configuration.
* I am passing through one GPU and one USB controller for each of two
Windows 10 guests (my Linux guests are running without passthrough at
this time, although also I experimented with such setup)
* kernel 4.1.17 (vanilla, no patches at all)
* useful bits in /etc :
bronek at gdansk /etc % cat modprobe.d/blacklist.conf
blacklist nouveau
blacklist snd_hda_intel
bronek at gdansk /etc % cat modprobe.d/kvm.conf
options kvm ignore_msrs=1
bronek at gdansk /etc % cat modprobe.d/vfio.conf
options vfio-pci ids=1912:0014,10de:13f0,10de:0fbb
bronek at gdansk /etc % cat sysctl.d/80-hugepages.conf
# Reserve this many 2MB pages for virtual machine
vm.nr_hugepages = 28000
bronek at gdansk /etc % grep -E "^MODULES" /etc/mkinitcpio.conf
MODULES="vfio vfio_iommu_type1 vfio_pci vfio_virqfd vhost_net bridge
mpt2sas nvme"
bronek at gdansk /etc % sed 's|192.168.[0-9.]*|192.168.#.#|g' netctl/br0
Description="Bridge connection"
Interface=br0
Connection=bridge
BindsToInterfaces=(enp5s0f1)
IP=static
Address=('192.168.#.#/24')
## Do not want default gateway on this interface
# Gateway='192.168.#.#'
Routes=('192.168.#.#/24 via 192.168.#.#')
DNS=('192.168.#.#' '192.168.#.#')
NETCTL_DEBUG=yes
## Ignore (R)STP and immediately activate the bridge
SkbpForwardingDelay=yes
bronek at gdansk /etc % lspci -tvnn | less
. . .
+-[0000:80]-+-01.0-[81]--
| +-02.0-[82]--+-00.0 NVIDIA Corporation GM204GL [Quadro
M5000] [10de:13f0]
| | \-00.1 NVIDIA Corporation GM204 High
Definition Audio Controller [10de:0fbb]
| +-03.0-[83]----00.0 Intel Corporation PCIe Data Center
SSD [8086:0953]
| +-03.2-[84]----00.0 Renesas Technology Corp. uPD720201
USB 3.0 Host Controller [1912:0014]
. . .
\-[0000:00]-+-00.0 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7
DMI2 [8086:0e00]
+-01.0-[01]----00.0 LSI Logic / Symbios Logic SAS2308
PCI-Express Fusion-MPT SAS-2 [1000:0086]
+-02.0-[02]----00.0 Renesas Technology Corp. uPD720201
USB 3.0 Host Controller [1912:0014]
+-03.0-[03]--+-00.0 NVIDIA Corporation GM204GL [Quadro
M5000] [10de:13f0]
| \-00.1 NVIDIA Corporation GM204 High
Definition Audio Controller [10de:0fbb]
* boot configuration (i.e. kernel command line parameters):
console=ttyS0,115200N8R nomodest nohz=off udev.children-max=32
edac_core.edac_mc_panic_on_ue=1 intel_iommu=on iommu=pt
isolcpus=4,5,6,7,12,13,14,15,20,21,22,23,28,29,30,31
(I removed parts relevant to ZFS only, which I also use)
* my guests are using UEFI boot in kvm (my host is starting from
"legacy" BIOS, though), using edk2.git-ovmf-x64-... from
https://www.kraxel.org/repos/jenkins/edk2/
* for Windows installation (and possibnly boot-level maintenance in the
future) I use libvirt profile like attached "-spice" profile. I start
and connect to this from an old laptop running Linux.
* also, as you will see in attached profiles, I am using virtio-scsi.
This means that at the start of Windows setup I need to install vioscsi
driver from virtio-win-0.1.112.iso (because otherwise installer won't
see disk to install to)
* first thing after Windows boot (still under spice), I install
remaining drivers and qemu guest-agent from virtio-win-0.1.112.iso
(note, use qxldod rather than qxl)
* when I'm done setting up Windows (still under spice), I issue two
commands to attach both passed-through cards and install drivers:
# attach-device --live bochnia-spice vfio2-nec.xml
# attach-device --live bochnia-spice vfio2-nvidia.xml
* after this I shutdown virtual machine running "-spice" profile and
then start it again using attached "-vfio2" profile i.e. replaced spice
with passed-through hardware
* NOTE: this line "<rom bar='off'/>" in attached vfio2 - without it my
passed through Quadro will not work at all.
When I'm done, I keep "spice" profile defined in libvirt "just in case",
but have not used it so far at all. Passed through card just works, lack
of Windows boot screen aside. FWIW, attached vfio2 is translated to this
command line by libvirt:
root at gdansk ~ # ps lfx | grep bochnia | grep -v grep
6 0 7494 1 20 0 20843916 90292 poll_s SLl ? 217:53
/usr/bin/qemu-system-x86_64 -name
bochnia-vfio2,process=qemu:bochnia-vfio2 -S -machine
pc-i440fx-2.5,accel=kvm,usb=off,mem-merge=off -cpu
host,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1000 -drive
file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd,if=pflash,format=raw,unit=0,readonly=on
-drive
file=/var/lib/libvirt/qemu/nvram/bochnia-spice_VARS.fd,if=pflash,format=raw,unit=1
-m 16384 -mem-prealloc -mem-path /dev/hugepages/libvirt/qemu -realtime
mlock=off -smp 8,sockets=1,cores=4,threads=2 -uuid
dd4d4b52-0e06-41da-8357-c7792988ec4f -nographic -no-user-config
-nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-bochnia-vfio2/monitor.sock,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc
base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet
-no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1
-boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7
-device
ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6
-device
ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1
-device
ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2
-device virtio-scsi-pci,id=scsi0,num_queues=8,bus=pci.0,addr=0x4 -device
virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive
file=/dev/zvol/zdata/vdis/bochnia,format=raw,if=none,id=drive-scsi0-0-0-0,cache=writeback
-device
scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1
-drive
file=/dev/disk/by-partuuid/099c0f43-a887-4892-b459-89d0535739fa,format=raw,if=none,id=drive-scsi0-0-1-0,cache=none
-device
scsi-hd,bus=scsi0.0,channel=0,scsi-id=1,lun=0,drive=drive-scsi0-0-1-0,id=scsi0-0-1-0
-drive
file=/data/vdis/isos/Windows10_x64_en_GB.iso,format=raw,if=none,id=drive-ide0-0-0,readonly=on
-device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -drive
file=/data/vdis/isos/virtio-win-0.1.112.iso,format=raw,if=none,id=drive-ide0-0-1,readonly=on
-device ide-cd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1 -netdev
tap,fds=25:27:28:29:30:31:32:33,id=hostnet0,vhost=on,vhostfds=34:35:36:37:38:39:40:41
-device
virtio-net-pci,mq=on,vectors=18,netdev=hostnet0,id=net0,mac=52:54:01:34:2e:f3,bus=pci.0,addr=0x18
-chardev
socket,id=charchannel0,path=/var/lib/libvirt/qemu/bochnia.agent,server,nowait
-device
virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0
-device usb-host,hostbus=4,hostaddr=4,id=hostdev0 -device
vfio-pci,host=84:00.0,id=hostdev1,bus=pci.0,addr=0x9 -device
vfio-pci,host=82:00.0,id=hostdev2,bus=pci.0,addr=0x8,rombar=0 -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on
B.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: vfio2-nec.xml
Type: text/xml
Size: 286 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20160224/22937cb0/attachment.xml>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: vfio2-nvidia.xml
Type: text/xml
Size: 308 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20160224/22937cb0/attachment-0001.xml>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: bochnia-spice.xml
Type: text/xml
Size: 5270 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20160224/22937cb0/attachment-0002.xml>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: bochnia-vfio2.xml
Type: text/xml
Size: 5545 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20160224/22937cb0/attachment-0003.xml>
More information about the vfio-users
mailing list