[vfio-users] Trying to get VFIO working on my laptop

[Daurnimator]

Hi all,

I'm trying to get windows working inside QEMU on my laptop with
passthrough of my gfx card.
However, I seem to be running getting the infamous code 43 from the
nvidia driver.
Screenshot of device manager: https://i.imgur.com/YA2baaW.png
The solutions I've found online so far don't seem to help, so I'm
posting to this list.

The following is a well commented shell script of how I've progressed
so far (latest available at
https://gist.github.com/daurnimator/01ae36e54eeb46668696468f18c1bfef):

# About

## Hardware
### Dell Precision M3800 laptop
#### Quad-core processor. Supports VT-d
#### 16GB RAM
#### Intel onboard GFX
#### GPU: 02:00.0 3D controller [0302]: NVIDIA Corporation GK107GLM
[Quadro K1100M] [10de:0ff6]

## Kernel
### Linux 4.5.1-1-ARCH #1 SMP PREEMPT Thu Apr 14 19:19:32 CEST 2016
x86_64 GNU/Linux


# Setup

## Make sure Virtualisation is turned on in BIOS

## Turn on IOMMU
### Need to add relevant iommu option to your kernel command line
### e.g. in /etc/defaults/grub
### For intel processors, add `intel_iommu=on`
### This resulted in a heap of errors in my kernel log like:
### 'DMAR: DMAR:[DMA Read] Request device [00:02.0] fault addr 2927799'
### Fixed this by adding igfx_off. so a total of: `intel_iommu=on,igfx_off`

## Permissions
### add udev rule to tag vfio device with 'kvm' group
### add self to kvm group
### Related: http://www.evonide.com/non-root-gpu-passthrough-setup/

## I got error 'No available IOMMU models'. Had to run:
### Found fix via: https://bugs.launchpad.net/qemu/+bug/1529187
sudo modprobe vfio_iommu_type1

## I got error 'VFIO_MAP_DMA cannot allocate memory'
### need to increase locked memory ulimit
### https://bugzilla.redhat.com/show_bug.cgi?id=912277
### edited /etc/security/limits.conf, allowed 10GB (I guess it needs
to be more than GFX card RAM?):
### daurnimator     hard    memlock     10485760
ulimit -l 10485760

## I get error 'qemu-system-x86_64: vfio-pci: Cannot read device rom
at 0000:02:00.0'
### In kernel log: 'vfio-pci 0000:02:00.0: Invalid PCI ROM header
signature: expecting 0xaa55, got 0x0000'


# Assemble qemu args

## Q35 is a good base chipset, run `qemu-system-x86_64 -machine help`
to get a list
opts="$opts -machine type=q35,accel=kvm"
## KVM allows for faster guests
opts="$opts -enable-kvm"
## Guest gets same CPU model as host (only works with KVM enabled).
### Don't let guest use KVM itself. Also apparently solves an issue:
#### From https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#QEMU_commands:
#### 'kvm=off is used for NVIDIA cards to stop it detecting a
hypervisor and therefore exiting with an error.'
#opts="$opts -cpu host,kvm=off"
### Fix from http://vfio.blogspot.com/2014/08/vfiovga-faq.html?showComment=1452870650776#c3703276360523701353
opts="$opts -cpu
host,kvm=off,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,hv_vendor_id=Nvidia43FIX"
## I have a quad core processor. let guest have 6 CPUs (3 cores with
hyperthreading)
opts="$opts -smp 6,sockets=1,cores=3,threads=2"
## I have 16GB of RAM, let guest have 8GB.
opts="$opts -m 8G"
## Give guest an emulated intel sound card
opts="$opts -soundhw hda"
## Give guest video card.
### Need to add the vendor + product to /etc/modprobe.d/vfio.cfg
### Pass PCI Bridge the GFX card is on.
### XXX: Where are docs on multifunction and x-vga options??
#### Info on x-vga:
http://www.firewing1.com/howtos/fedora-20/create-gaming-virtual-machine-using-vfio-pci-passthrough-kvm
#opts="$opts -device vfio-pci,host=00:01.0,multifunction=on,x-vga=on"
### Then the GFX card itself
#### If you pass x-vga=on it complains: 'vfio: Device does not support
requested feature x-vga'
opts="$opts -device vfio-pci,host=02:00.0,multifunction=on"

## Hard Drive
### Created with 'qemu-img create -f raw win.img 16G' then installed
windows onto it
opts="$opts -drive file=win.img,format=raw"

# Start Qemu
exec qemu-system-x86_64 $opts $*