<div dir="ltr"><div><div><div><div>Your luck, i've got network connectivity with my machine.<br></div>So, here's my full config, as it works for me. That's one of the very few combinations that really work, not resulting in code-10, hard freeze or host shutdown.<br><br></div><div>First of all, lspci of the used devices:<br>00:14.2 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] FCH Azalia Controller [1022:780d] (rev 01)<br>...<br>01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde PRO [Radeon HD 7750/8740 / R7 250E] [1002:683f]<br>01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] [1002:aab0]<br>02:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde PRO [Radeon HD 7750/8740 / R7 250E] [1002:683f]<br>02:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] [1002:aab0]<br>03:05.0 Multimedia audio controller [0401]: Ensoniq 5880B [AudioPCI] [1274:5880] (rev 02)<br>...<br>07:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF119 [GeForce GT 610] [10de:104a] (rev a1)<br>07:00.1 Audio device [0403]: NVIDIA Corporation GF119 HDMI Audio Controller [10de:0e08] (rev a1)<br></div><div>The third GPU(nvidia) is connected to PCI-E x1, any of two. When connected to the upper, it becomes 04:00.0, when the lower one is used - 07:00.1<br></div><div>It may also be connected to the x16 slot and everything will work fine. Except you have to tell NVidia proprietary drivers where the GPU is located, adding BusID to xorg.conf:<br></div><div>FILE: /etc/X11/xorg.conf<br>Section "Device"<br> Identifier "Device0"<br> Driver "nvidia"<br> VendorName "NVIDIA Corporation"<br> BoardName "GeForce GT 610"<br> BusID "PCI:07:00:00"<br>EndSection<br></div><div><br>General info:<br>[user@crossfire ~]$ uname -a<br>Linux crossfire 4.1.5-200.fc22.x86_64 #1 SMP Mon Aug 10 23:38:23 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux<br>[user@crossfire ~]$ qemu-system-x86_64 --version<br>QEMU emulator version 2.3.1 (qemu-2.3.1-1.fc22), Copyright (c) 2003-2008 Fabrice Bellard<br>[user@crossfire ~]$ dnf info edk2.git-ovmf-x64<br>Installed Packages<br>Name : edk2.git-ovmf-x64<br>Arch : noarch<br>Epoch : 0<br>Version : 0<br>Release : 20150804.b1143.g8ca1489<br>Size : 11 M<br>Repo : @System<br>From repo : qemu-firmware-jenkins<br></div>(newer versions work too)<br><br>FILE: /etc/modprobe.d/kvm-amd.conf<br>options kvm-amd npt=1 nested=0<br>options kvm ignore_msrs=0 allow_unsafe_assigned_interrupts=0<br>options vfio_iommu_type1 disable_hugepages=1 allow_unsafe_interrupts=0<br></div><br>Note disabled hugepages and nothing else much enabled. Toggling NPT off may give you a performance boost(while being counter-intuitive), toggling nested on may allow you to use stuff like Hyper-V in the VM, i couldn't get it working right.<br></div>Radeon drivers may be blacklisted. I don't find it necessary for me.<br><div><br>FILE: /etc/default/grub<br>GRUB_CMDLINE_LINUX="vconsole.font=latarcyrheb-sun16 $([ -x /usr/sbin/rhcrashkernel-param ] && /usr/sbin/rhcrashkernel-param || :) nofb pci-stub.ids=1022:780d,1002:683f,1002:aab0 enable_mtrr_cleanup amd_iommu_dump=1 iommu=pt acpi_enforce_resources=lax<br><br><div>Note that i am using the old way of isolating devices from the system, by appending PCI IDs to the pci-stub.<br></div><div>MTRR cleanup is not so necessary, AMD_IOMMU_DUMP is providing sometimes useful stuff in dmesg so can be safely disabled.<br></div><div>IOMMU=pt isn't necessary.<br></div><div>ACPI_enforce_resources=lax... i don't remember, try without it first.<br><br></div><div>Anyway, that's the easy part of configurating stuff.<br></div><div>Now the libvirt domain XML:<br> <os><br> <type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type><br> <loader readonly='yes' type='pflash'>/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd</loader><br> <nvram>/var/lib/libvirt/qemu/nvram/vm1_VARS.fd</nvram><br> <bootmenu enable='no'/><br> </os><br> <features><br> <acpi/><br> <apic/><br> <pae/><br> <hyperv><br> <relaxed state='on'/><br> <vapic state='on'/><br> <spinlocks state='on' retries='8191'/><br> </hyperv><br> </features><br><cpu mode='host-model'><br> <model fallback='allow'/><br> <topology sockets='1' cores='4' threads='1'/><br> </cpu><br> <clock offset='localtime'><br> <timer name='rtc' tickpolicy='catchup'/><br> <timer name='pit' tickpolicy='delay'/><br> <timer name='hpet' present='no'/><br> <timer name='hypervclock' present='yes'/><br> </clock><br></div><div>Note the default way of using OVMF and enabled Hyper-V(I didn't try to pass-through my GT610, so you might want to disable this if you use NVidia).<br>...<br> <video><br> <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1'/><br> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/><br> </video><br></div><div>QXL is used just for windows 7 as noted in the OVMF whitepaper - this special device contains a shim for windows7's use of VBE and translates that to GOP, making Windows 7 more UEFI capable. This device can be safely thrown out if using Win8.1 or Win10.<br><br></div><div><hostdev mode='subsystem' type='pci' managed='yes'><br> <source><br> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/><br> </source><br> <rom bar='on' file='/mnt/hdd/qemu/hybridmagic.rom'/><br> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/><br> </hostdev><br> <hostdev mode='subsystem' type='pci' managed='yes'><br> <source><br> <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/><br> </source><br> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/><br> </hostdev><br> <hostdev mode='subsystem' type='pci' managed='yes'><br> <source><br> <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/><br> </source><br> <rom bar='on' file='/mnt/hdd/qemu/hybridmagic.rom'/><br> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/><br> </hostdev><br> <hostdev mode='subsystem' type='pci' managed='yes'><br> <source><br> <address domain='0x0000' bus='0x02' slot='0x00' function='0x1'/><br> </source><br> <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/><br> </hostdev><br> <hostdev mode='subsystem' type='pci' managed='yes'><br> <driver name='kvm'/><br> <source><br> <address domain='0x0000' bus='0x00' slot='0x14' function='0x2'/><br> </source><br> <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/><br> </hostdev><br></div><div>Note the ROM files appended to the GPUs - i had to glue the GOP part from HD6450 and my GPU's VBIOS together to get this magic file. It is only needed if the VM locks up to 100% CPU when UEFI is used with your GPU.<br></div><div>Also note that the audio device i am passing through is using legacy pci-assign since it has bad IOMMU grouping, but since this is an audio card, it's fine with that. Can be safely thrown out.<br><br></div><div>Please warn me if you need generic qemu command-line options used to get it working, i have those too.<br><br></div><div>As you can see, it's a real pain in the butt to get it all working.<br></div><div><br></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">2015-09-07 13:15 GMT+03:00 Blank Field <span dir="ltr"><<a href="mailto:ihatethisfield@gmail.com" target="_blank">ihatethisfield@gmail.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><p dir="ltr">AMD FM2 platform is broken as hell: i have F2A55 and it took me a year to figure out all the workarounds.<br> So, try using OVMF, without hugepages and VGA.<br> It works for me, i can share my configuration with you once i get to the machine.</p> <div class="gmail_quote">On Sep 7, 2015 12:22 PM, "Aaron Lancaster" <<a href="mailto:aaronlancaster74@gmail.com" target="_blank">aaronlancaster74@gmail.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I am attempting to do PCI redirection with KVM on a GA-F2A88XM-D3H motherboard.<br> <br> I can redirect every other device/slot on the system except the PCI-E<br> x16 slot that contains the card the UEFI/BIOS displays on.<br> <br> If I attempt to redirect a card in this slot I immediately get a full<br> system freeze where nothing responds. I have two PCI-E graphics cards,<br> and this happens with both of them, and both of them can be redirected<br> without issue when placed into the secondary PCI-E 4x slot.<br> <br> I have ensured that vfio-pci is bound to the PCI device after boot<br> (via initrd), and that everything in the iommu group is being<br> redirected.<br> <br> Anything i may have missed, or some tips?<br> <br> _______________________________________________<br> vfio-users mailing list<br> <a href="mailto:vfio-users@redhat.com" target="_blank">vfio-users@redhat.com</a><br> <a href="https://www.redhat.com/mailman/listinfo/vfio-users" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/vfio-users</a><br> </blockquote></div> </blockquote></div><br></div>