[vfio-users] VFIO PCI-E redirection hardlock
Blank Field
ihatethisfield at gmail.com
Mon Sep 7 10:42:54 UTC 2015
Your luck, i've got network connectivity with my machine.
So, here's my full config, as it works for me. That's one of the very few
combinations that really work, not resulting in code-10, hard freeze or
host shutdown.
First of all, lspci of the used devices:
00:14.2 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] FCH Azalia
Controller [1022:780d] (rev 01)
...
01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc.
[AMD/ATI] Cape Verde PRO [Radeon HD 7750/8740 / R7 250E] [1002:683f]
01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cape
Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] [1002:aab0]
02:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc.
[AMD/ATI] Cape Verde PRO [Radeon HD 7750/8740 / R7 250E] [1002:683f]
02:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cape
Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] [1002:aab0]
03:05.0 Multimedia audio controller [0401]: Ensoniq 5880B [AudioPCI]
[1274:5880] (rev 02)
...
07:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF119 [GeForce
GT 610] [10de:104a] (rev a1)
07:00.1 Audio device [0403]: NVIDIA Corporation GF119 HDMI Audio Controller
[10de:0e08] (rev a1)
The third GPU(nvidia) is connected to PCI-E x1, any of two. When connected
to the upper, it becomes 04:00.0, when the lower one is used - 07:00.1
It may also be connected to the x16 slot and everything will work fine.
Except you have to tell NVidia proprietary drivers where the GPU is
located, adding BusID to xorg.conf:
FILE: /etc/X11/xorg.conf
Section "Device"
Identifier "Device0"
Driver "nvidia"
VendorName "NVIDIA Corporation"
BoardName "GeForce GT 610"
BusID "PCI:07:00:00"
EndSection
General info:
[user at crossfire ~]$ uname -a
Linux crossfire 4.1.5-200.fc22.x86_64 #1 SMP Mon Aug 10 23:38:23 UTC 2015
x86_64 x86_64 x86_64 GNU/Linux
[user at crossfire ~]$ qemu-system-x86_64 --version
QEMU emulator version 2.3.1 (qemu-2.3.1-1.fc22), Copyright (c) 2003-2008
Fabrice Bellard
[user at crossfire ~]$ dnf info edk2.git-ovmf-x64
Installed Packages
Name : edk2.git-ovmf-x64
Arch : noarch
Epoch : 0
Version : 0
Release : 20150804.b1143.g8ca1489
Size : 11 M
Repo : @System
>From repo : qemu-firmware-jenkins
(newer versions work too)
FILE: /etc/modprobe.d/kvm-amd.conf
options kvm-amd npt=1 nested=0
options kvm ignore_msrs=0 allow_unsafe_assigned_interrupts=0
options vfio_iommu_type1 disable_hugepages=1 allow_unsafe_interrupts=0
Note disabled hugepages and nothing else much enabled. Toggling NPT off may
give you a performance boost(while being counter-intuitive), toggling
nested on may allow you to use stuff like Hyper-V in the VM, i couldn't get
it working right.
Radeon drivers may be blacklisted. I don't find it necessary for me.
FILE: /etc/default/grub
GRUB_CMDLINE_LINUX="vconsole.font=latarcyrheb-sun16 $([ -x
/usr/sbin/rhcrashkernel-param ] && /usr/sbin/rhcrashkernel-param || :) nofb
pci-stub.ids=1022:780d,1002:683f,1002:aab0 enable_mtrr_cleanup
amd_iommu_dump=1 iommu=pt acpi_enforce_resources=lax
Note that i am using the old way of isolating devices from the system, by
appending PCI IDs to the pci-stub.
MTRR cleanup is not so necessary, AMD_IOMMU_DUMP is providing sometimes
useful stuff in dmesg so can be safely disabled.
IOMMU=pt isn't necessary.
ACPI_enforce_resources=lax... i don't remember, try without it first.
Anyway, that's the easy part of configurating stuff.
Now the libvirt domain XML:
<os>
<type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type>
<loader readonly='yes'
type='pflash'>/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
<nvram>/var/lib/libvirt/qemu/nvram/vm1_VARS.fd</nvram>
<bootmenu enable='no'/>
</os>
<features>
<acpi/>
<apic/>
<pae/>
<hyperv>
<relaxed state='on'/>
<vapic state='on'/>
<spinlocks state='on' retries='8191'/>
</hyperv>
</features>
<cpu mode='host-model'>
<model fallback='allow'/>
<topology sockets='1' cores='4' threads='1'/>
</cpu>
<clock offset='localtime'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
<timer name='hypervclock' present='yes'/>
</clock>
Note the default way of using OVMF and enabled Hyper-V(I didn't try to
pass-through my GT610, so you might want to disable this if you use NVidia).
...
<video>
<model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04'
function='0x0'/>
</video>
QXL is used just for windows 7 as noted in the OVMF whitepaper - this
special device contains a shim for windows7's use of VBE and translates
that to GOP, making Windows 7 more UEFI capable. This device can be safely
thrown out if using Win8.1 or Win10.
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</source>
<rom bar='on' file='/mnt/hdd/qemu/hybridmagic.rom'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03'
function='0x0'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
</source>
<address type='pci' domain='0x0000' bus='0x00' slot='0x08'
function='0x0'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
</source>
<rom bar='on' file='/mnt/hdd/qemu/hybridmagic.rom'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09'
function='0x0'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x02' slot='0x00' function='0x1'/>
</source>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0a'
function='0x0'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='kvm'/>
<source>
<address domain='0x0000' bus='0x00' slot='0x14' function='0x2'/>
</source>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0b'
function='0x0'/>
</hostdev>
Note the ROM files appended to the GPUs - i had to glue the GOP part from
HD6450 and my GPU's VBIOS together to get this magic file. It is only
needed if the VM locks up to 100% CPU when UEFI is used with your GPU.
Also note that the audio device i am passing through is using legacy
pci-assign since it has bad IOMMU grouping, but since this is an audio
card, it's fine with that. Can be safely thrown out.
Please warn me if you need generic qemu command-line options used to get it
working, i have those too.
As you can see, it's a real pain in the butt to get it all working.
2015-09-07 13:15 GMT+03:00 Blank Field <ihatethisfield at gmail.com>:
> AMD FM2 platform is broken as hell: i have F2A55 and it took me a year to
> figure out all the workarounds.
> So, try using OVMF, without hugepages and VGA.
> It works for me, i can share my configuration with you once i get to the
> machine.
> On Sep 7, 2015 12:22 PM, "Aaron Lancaster" <aaronlancaster74 at gmail.com>
> wrote:
>
>> I am attempting to do PCI redirection with KVM on a GA-F2A88XM-D3H
>> motherboard.
>>
>> I can redirect every other device/slot on the system except the PCI-E
>> x16 slot that contains the card the UEFI/BIOS displays on.
>>
>> If I attempt to redirect a card in this slot I immediately get a full
>> system freeze where nothing responds. I have two PCI-E graphics cards,
>> and this happens with both of them, and both of them can be redirected
>> without issue when placed into the secondary PCI-E 4x slot.
>>
>> I have ensured that vfio-pci is bound to the PCI device after boot
>> (via initrd), and that everything in the iommu group is being
>> redirected.
>>
>> Anything i may have missed, or some tips?
>>
>> _______________________________________________
>> vfio-users mailing list
>> vfio-users at redhat.com
>> https://www.redhat.com/mailman/listinfo/vfio-users
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20150907/d89fc661/attachment.htm>
More information about the vfio-users
mailing list