[vfio-users] vfio PCI passthrough NVIDIA K420 GPU problems

Wed Feb 24 20:44:23 UTC 2016

On 24/02/2016 20:26, Bronek Kozicki wrote:
> On 24/02/2016 15:41, Daniel Pocock wrote:
>>
>>
>> Hi,
>>
>> I'm trying to use PCI passthrough to give an NVIDIA GPU to a VM with
>> qemu / KVM.  I've summarized my environment below and the error I get is
>> near the bottom.  Any help would be appreciated.
>>
>> There are a few guides I've been referring to already:
>> https://wiki.debian.org/VGAPassthrough
>> https://www.pugetsystems.com/labs/articles/Multiheaded-NVIDIA-Gaming-using-Ubuntu-14-04-KVM-585/
>>
>> https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF
>> https://bbs.archlinux.org/viewtopic.php?id=162768
>> http://www.linux-kvm.org/page/VGA_device_assignment
>
>
> Hi Daniel
>
>
> I'm successfully passing through two Quadro M5000 (to two instances of
> Windows 10), looking at
> http://us.download.nvidia.com/Windows/Quadro_Certified/361.91/361.91-win10-quadro-grid-release-notes.pdf
> (page 9) it is not obvious that this would work for either your K420 or
> mine M5000. One gotcha - when starting Windows I do not see boot screen
> at all, only Windows logon scren after nVidia drivers had loaded.
> Explanation is in Alex's email sent to this list on 6th Feb 2016,
> subject "No boot screen on Quadro M5000?" - basically passed through
> Quadro cards are meant to be secondary  only (I use them as only card,
> though). However this small quirk aside, this works for me very well.
> Here are details of my setup:
>
> * distribution Arch , with packages : qemu 2.5.0 libvirt 1.3.1 (I do not
> use qemu command line directly)
>
> * my machine has 32 logical cores (2 x 8 core Xeon E5-26xx CPUs, with
> HT) and 128GB ECC RAM, two Quadro M5000 cards and two USB 3.0 controller
> cards; my motherboard is Supermicro X9DA7. I also dedicated half of
> cores and about half of RAM to guests (I'm also running Linux virtual
> machines with spice). Obviously you do not need this much hardware, just
> a context to help you understand my configuration.
>
> * I am passing through one GPU and one USB controller for each of two
> Windows 10 guests (my Linux guests are running without passthrough at
> this time, although also I experimented with such setup)
>
> * kernel 4.1.17 (vanilla, no patches at all)
>
> * useful bits in /etc :
>    bronek at gdansk /etc % cat modprobe.d/blacklist.conf
>    blacklist nouveau
>    blacklist snd_hda_intel
>
>    bronek at gdansk /etc % cat modprobe.d/kvm.conf
>    options kvm ignore_msrs=1
>
>    bronek at gdansk /etc % cat modprobe.d/vfio.conf
>    options vfio-pci ids=1912:0014,10de:13f0,10de:0fbb
>
>    bronek at gdansk /etc % cat sysctl.d/80-hugepages.conf
>    # Reserve this many 2MB pages for virtual machine
>    vm.nr_hugepages = 28000
>
>    bronek at gdansk /etc % grep -E "^MODULES" /etc/mkinitcpio.conf
>    MODULES="vfio vfio_iommu_type1 vfio_pci vfio_virqfd vhost_net bridge
> mpt2sas nvme"
>
>    bronek at gdansk /etc % sed 's|192.168.[0-9.]*|192.168.#.#|g' netctl/br0
>    Description="Bridge connection"
>    Interface=br0
>    Connection=bridge
>    BindsToInterfaces=(enp5s0f1)
>    IP=static
>    Address=('192.168.#.#/24')
>    ## Do not want default gateway on this interface
>    # Gateway='192.168.#.#'
>    Routes=('192.168.#.#/24 via 192.168.#.#')
>    DNS=('192.168.#.#' '192.168.#.#')
>    NETCTL_DEBUG=yes
>    ## Ignore (R)STP and immediately activate the bridge
>    SkbpForwardingDelay=yes
>
>    bronek at gdansk /etc % lspci -tvnn | less
>    . . .
>   +-[0000:80]-+-01.0-[81]--
>   |           +-02.0-[82]--+-00.0  NVIDIA Corporation GM204GL [Quadro
> M5000] [10de:13f0]
>   |           |            \-00.1  NVIDIA Corporation GM204 High
> Definition Audio Controller [10de:0fbb]
>   |           +-03.0-[83]----00.0  Intel Corporation PCIe Data Center
> SSD [8086:0953]
>   |           +-03.2-[84]----00.0  Renesas Technology Corp. uPD720201
> USB 3.0 Host Controller [1912:0014]
>    . . .
>   \-[0000:00]-+-00.0  Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7
> DMI2 [8086:0e00]
>               +-01.0-[01]----00.0  LSI Logic / Symbios Logic SAS2308
> PCI-Express Fusion-MPT SAS-2 [1000:0086]
>               +-02.0-[02]----00.0  Renesas Technology Corp. uPD720201
> USB 3.0 Host Controller [1912:0014]
>               +-03.0-[03]--+-00.0  NVIDIA Corporation GM204GL [Quadro
> M5000] [10de:13f0]
>               |            \-00.1  NVIDIA Corporation GM204 High
> Definition Audio Controller [10de:0fbb]
>
> * boot configuration (i.e. kernel command line parameters):
> console=ttyS0,115200N8R nomodest nohz=off udev.children-max=32
> edac_core.edac_mc_panic_on_ue=1 intel_iommu=on iommu=pt
> isolcpus=4,5,6,7,12,13,14,15,20,21,22,23,28,29,30,31
> (I removed parts relevant to ZFS only, which I also use)
>
> * my guests are using UEFI boot in kvm (my host is starting from
> "legacy" BIOS, though), using edk2.git-ovmf-x64-... from
> https://www.kraxel.org/repos/jenkins/edk2/
>
> * for Windows installation (and possibnly boot-level maintenance in the
> future) I use libvirt profile like attached "-spice" profile. I start
> and connect to this from an old laptop running Linux.
>
> * also, as you will see in attached profiles, I am using virtio-scsi.
> This means that at the start of Windows setup I need to install vioscsi
> driver from virtio-win-0.1.112.iso (because otherwise installer won't
> see disk to install to)
>
> * first thing after Windows boot (still under spice), I install
> remaining drivers and qemu guest-agent from virtio-win-0.1.112.iso
> (note, use qxldod rather than qxl)
>
> * when I'm done setting up Windows (still under spice), I issue two
> commands to attach both passed-through cards and install drivers:
> # attach-device --live bochnia-spice vfio2-nec.xml
> # attach-device --live bochnia-spice vfio2-nvidia.xml
>
> * after this I shutdown virtual machine running "-spice" profile and
> then start it again using attached "-vfio2" profile i.e. replaced spice
> with passed-through hardware
>
> * NOTE: this line "<rom bar='off'/>" in attached vfio2 - without it my
> passed through Quadro will not work at all.
>
> When I'm done, I keep "spice" profile defined in libvirt "just in case",
> but have not used it so far at all. Passed through card just works, lack
> of Windows boot screen aside. FWIW, attached vfio2 is translated to this
> command line by libvirt:
>
>
> root at gdansk ~ # ps lfx | grep bochnia | grep -v grep
> 6     0  7494     1  20   0 20843916 90292 poll_s SLl ?       217:53
> /usr/bin/qemu-system-x86_64 -name
> bochnia-vfio2,process=qemu:bochnia-vfio2 -S -machine
> pc-i440fx-2.5,accel=kvm,usb=off,mem-merge=off -cpu
> host,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1000 -drive
> file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd,if=pflash,format=raw,unit=0,readonly=on
> -drive
> file=/var/lib/libvirt/qemu/nvram/bochnia-spice_VARS.fd,if=pflash,format=raw,unit=1
> -m 16384 -mem-prealloc -mem-path /dev/hugepages/libvirt/qemu -realtime
> mlock=off -smp 8,sockets=1,cores=4,threads=2 -uuid
> dd4d4b52-0e06-41da-8357-c7792988ec4f -nographic -no-user-config
> -nodefaults -chardev
> socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-bochnia-vfio2/monitor.sock,server,nowait
> -mon chardev=charmonitor,id=monitor,mode=control -rtc
> base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet
> -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1
> -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7
> -device
> ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6
> -device
> ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1
> -device
> ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2
> -device virtio-scsi-pci,id=scsi0,num_queues=8,bus=pci.0,addr=0x4 -device
> virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive
> file=/dev/zvol/zdata/vdis/bochnia,format=raw,if=none,id=drive-scsi0-0-0-0,cache=writeback
> -device
> scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1
> -drive
> file=/dev/disk/by-partuuid/099c0f43-a887-4892-b459-89d0535739fa,format=raw,if=none,id=drive-scsi0-0-1-0,cache=none
> -device
> scsi-hd,bus=scsi0.0,channel=0,scsi-id=1,lun=0,drive=drive-scsi0-0-1-0,id=scsi0-0-1-0
> -drive
> file=/data/vdis/isos/Windows10_x64_en_GB.iso,format=raw,if=none,id=drive-ide0-0-0,readonly=on
> -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -drive
> file=/data/vdis/isos/virtio-win-0.1.112.iso,format=raw,if=none,id=drive-ide0-0-1,readonly=on
> -device ide-cd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1 -netdev
> tap,fds=25:27:28:29:30:31:32:33,id=hostnet0,vhost=on,vhostfds=34:35:36:37:38:39:40:41
> -device
> virtio-net-pci,mq=on,vectors=18,netdev=hostnet0,id=net0,mac=52:54:01:34:2e:f3,bus=pci.0,addr=0x18
> -chardev
> socket,id=charchannel0,path=/var/lib/libvirt/qemu/bochnia.agent,server,nowait
> -device
> virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0
> -device usb-host,hostbus=4,hostaddr=4,id=hostdev0 -device
> vfio-pci,host=84:00.0,id=hostdev1,bus=pci.0,addr=0x9 -device
> vfio-pci,host=82:00.0,id=hostdev2,bus=pci.0,addr=0x8,rombar=0 -device
> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on
>
>
>
>
> B.
>
>
>
>
>
> _______________________________________________
> vfio-users mailing list
> vfio-users at redhat.com
> https://www.redhat.com/mailman/listinfo/vfio-users
>