[vfio-users] Trying to get VFIO working on my laptop

Mon May 9 16:21:21 UTC 2016

On Mon, May 9, 2016 at 9:35 AM, Daurnimator <quae at daurnimator.com> wrote:

> Hi all,
>

Hi Daurnimator,

> I'm trying to get windows working inside QEMU on my laptop with
> passthrough of my gfx card.
> However, I seem to be running getting the infamous code 43 from the
> nvidia driver.
> Screenshot of device manager: https://i.imgur.com/YA2baaW.png
> The solutions I've found online so far don't seem to help, so I'm
> posting to this list.
>

In general Quadro cards passthrough are officially supported by NVIDIA
(except for some specific older ones, Alex could help confirm this). So
probably some steps outlined on web articles, which are mostly passing
through GeForce cards, behaves differently on Quadro. For instance,
although GeForce cards are not officially supported, GeForce cards could be
passed through as primary display adapter for the guest, unlike Quadro
cards which are only supported as secondary display adapter for the guest.

> The following is a well commented shell script of how I've progressed
> so far (latest available at
> https://gist.github.com/daurnimator/01ae36e54eeb46668696468f18c1bfef):
>
> # About
>
> ## Hardware
> ### Dell Precision M3800 laptop
> #### Quad-core processor. Supports VT-d
> #### 16GB RAM
> #### Intel onboard GFX
> #### GPU: 02:00.0 3D controller [0302]: NVIDIA Corporation GK107GLM
> [Quadro K1100M] [10de:0ff6]
>
> ## Kernel
> ### Linux 4.5.1-1-ARCH #1 SMP PREEMPT Thu Apr 14 19:19:32 CEST 2016
> x86_64 GNU/Linux
>
>
> # Setup
>
> ## Make sure Virtualisation is turned on in BIOS
>
> ## Turn on IOMMU
> ### Need to add relevant iommu option to your kernel command line
> ### e.g. in /etc/defaults/grub
> ### For intel processors, add `intel_iommu=on`
> ### This resulted in a heap of errors in my kernel log like:
> ### 'DMAR: DMAR:[DMA Read] Request device [00:02.0] fault addr 2927799'
> ### Fixed this by adding igfx_off. so a total of: `intel_iommu=on,igfx_off`
>
> ## Permissions
> ### add udev rule to tag vfio device with 'kvm' group
> ### add self to kvm group
> ### Related: http://www.evonide.com/non-root-gpu-passthrough-setup/
>
> ## I got error 'No available IOMMU models'. Had to run:
> ### Found fix via: https://bugs.launchpad.net/qemu/+bug/1529187
> sudo modprobe vfio_iommu_type1
>

I had to put VFIO modules inside /etc/mkinitcpio.conf so it gets loaded
very early during boot sequence. Regenerate kernel image after updating the
config.
MODULES="vfio vfio_iommu_type1 vfio_pci vfio_virqfd"

In combination with that, I put my device IDs on /etc/vfio-pci.conf and
disables VGA access to only permit passing through GPU via OVMF.
options vfio-pci
ids=10de:17c8,10de:0fb0,10de:1287,10de:0e0f,1b21:1142,1912:0014
options vfio-pci disable_vga=1

## I got error 'VFIO_MAP_DMA cannot allocate memory'
> ### need to increase locked memory ulimit
> ### https://bugzilla.redhat.com/show_bug.cgi?id=912277
> ### edited /etc/security/limits.conf, allowed 10GB (I guess it needs
> to be more than GFX card RAM?):
> ### daurnimator     hard    memlock     10485760
> ulimit -l 10485760
>
> ## I get error 'qemu-system-x86_64: vfio-pci: Cannot read device rom
> at 0000:02:00.0'
> ### In kernel log: 'vfio-pci 0000:02:00.0: Invalid PCI ROM header
> signature: expecting 0xaa55, got 0x0000'
>
>
> # Assemble qemu args
>
> ## Q35 is a good base chipset, run `qemu-system-x86_64 -machine help`
> to get a list
> opts="$opts -machine type=q35,accel=kvm"
>

If it is not necessary, I would avoid using Q35 chipset and use i440FX
since the latter is simpler and performs the same as Q35 without the added
complexities in building up and attaching the virtual PCIe root ports. I
have mine like this:
-machine type=pc-i440fx-2.5,accel=kvm,usb=off,vmport=off,kernel_irqchip=on

> ## KVM allows for faster guests
> opts="$opts -enable-kvm"
> ## Guest gets same CPU model as host (only works with KVM enabled).
> ### Don't let guest use KVM itself. Also apparently solves an issue:
> #### From
> https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#QEMU_commands
> :
> #### 'kvm=off is used for NVIDIA cards to stop it detecting a
> hypervisor and therefore exiting with an error.'
> #opts="$opts -cpu host,kvm=off"
> ### Fix from
> http://vfio.blogspot.com/2014/08/vfiovga-faq.html?showComment=1452870650776#c3703276360523701353
> opts="$opts -cpu
>
> host,kvm=off,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,hv_vendor_id=Nvidia43FIX"
>

Some people here in the list recommends omiting the hv_vapic flag for
recent CPU (I think starting Ivy Bridge onward), since it has native APIC-V
features built-in to the chip. My complete CPU line is like this:
-cpu
host,kvm=off,hv_time,hv_relaxed,hv_spinlocks=0x1fff,hv_vpindex,hv_reset,hv_runtime,hv_crash,hv_vendor_id=freyja

> ## I have a quad core processor. let guest have 6 CPUs (3 cores with
> hyperthreading)
> opts="$opts -smp 6,sockets=1,cores=3,threads=2"
> ## I have 16GB of RAM, let guest have 8GB.
> opts="$opts -m 8G"
>

Enabling explicit hugepages as the guest's memory backing should improve
performance especially for VFIO passthrough case. Assuming you have already
enabled explicit hugepages on the host, you can specify it with:
-mem-path /dev/hugepages

> ## Give guest an emulated intel sound card
> opts="$opts -soundhw hda"
> ## Give guest video card.
> ### Need to add the vendor + product to /etc/modprobe.d/vfio.cfg
> ### Pass PCI Bridge the GFX card is on.
> ### XXX: Where are docs on multifunction and x-vga options??
> #### Info on x-vga:
>
> http://www.firewing1.com/howtos/fedora-20/create-gaming-virtual-machine-using-vfio-pci-passthrough-kvm
> #opts="$opts -device vfio-pci,host=00:01.0,multifunction=on,x-vga=on"
> ### Then the GFX card itself
> #### If you pass x-vga=on it complains: 'vfio: Device does not support
> requested feature x-vga'
> opts="$opts -device vfio-pci,host=02:00.0,multifunction=on"
>

I assume your guest is a Windows 10 guest which has full support for UEFI
system. I recommend you to reinstall Windows 10 in UEFI by passing OVMF
split images as the UEFI code and variables. UEFI-based guest do not need
the x-vga=on flag as it is a legacy (BIOS) way to passthrough GPU to
guests. My OVMF lines look like this:
# OVMF split image
OPTS="$OPTS -drive if=pflash,format=raw,file=$KVM_DIR/$VM_NAME
/ovmf/OVMF_CODE-pure-efi.fd,readonly"
OPTS="$OPTS -drive if=pflash,format=raw,file=$KVM_DIR/$VM_NAME
/ovmf/OVMF_VARS-pure-efi.fd"

## Hard Drive
> ### Created with 'qemu-img create -f raw win.img 16G' then installed
> windows onto it
> opts="$opts -drive file=win.img,format=raw"
>
> # Start Qemu
> exec qemu-system-x86_64 $opts $*
>

Best regards,
Okky Hendriansyah
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20160509/2f9c5d01/attachment.htm>