<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>Hi,</div><div><br></div><div>If you using Radeon RX with i440fx type vm, GPU attached to pcie-root-complex trick is maybe needed.</div><div><br></div><div>i440fx and recent AMD Crimson drivers<br></div><div><a href="https://www.redhat.com/archives/vfio-users/2016-April/msg00153.html">https://www.redhat.com/archives/vfio-users/2016-April/msg00153.html</a><br></div><div dir="ltr"><br></div>Remove your Radeon RX Vega 56 passthrough config from <hostdev>~~</hostdev>section and using libvirt "<qemu:arg>" option, some thing like this.</div><div dir="ltr"></div><div dir="ltr">
<pre class="gmail-ipsCode gmail-prettyprint gmail-lang-html gmail-prettyprinted"><span class="gmail-pln"> </span><span class="gmail-tag"><qemu:commandline></span><span class="gmail-pln">
</span><span class="gmail-tag"><qemu:arg</span><span class="gmail-pln"> </span><span class="gmail-atn">value</span><span class="gmail-pun">=</span><span class="gmail-atv">'-device'</span><span class="gmail-tag">/></span><span class="gmail-pln">
</span><span class="gmail-tag"><qemu:arg</span><span class="gmail-pln"> </span><span class="gmail-atn">value</span><span class="gmail-pun">=</span><span class="gmail-atv">'ioh3420,bus=pci.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'</span><span class="gmail-tag">/></span><span class="gmail-pln">
</span><span class="gmail-tag"><qemu:arg</span><span class="gmail-pln"> </span><span class="gmail-atn">value</span><span class="gmail-pun">=</span><span class="gmail-atv">'-device'</span><span class="gmail-tag">/></span><span class="gmail-pln">
</span><span class="gmail-tag"><qemu:arg</span><span class="gmail-pln"> </span><span class="gmail-atn">value</span><span class="gmail-pun">=</span><span class="gmail-atv">'vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,rombar=on'</span><span class="gmail-tag">/></span><span class="gmail-pln">
</span><span class="gmail-tag"><qemu:arg</span><span class="gmail-pln"> </span><span class="gmail-atn">value</span><span class="gmail-pun">=</span><span class="gmail-atv">'-device'</span><span class="gmail-tag">/></span><span class="gmail-pln">
</span><span class="gmail-tag"><qemu:arg</span><span class="gmail-pln"> </span><span class="gmail-atn">value</span><span class="gmail-pun">=</span><span class="gmail-atv">'vfio-pci,host=01:00.1,bus=root.1,addr=00.1'</span><span class="gmail-tag">/></span><span class="gmail-pln">
</span><span class="gmail-tag"></qemu:commandline></span></pre>
</div><div dir="ltr"><br></div><div>Or, Use Q35 type VM with Intel ioh3420 pcie-root-port. For example, see below.<br></div><div><br></div><div> <controller type='pci' index='2' model='pcie-root-port'><br> <model name='ioh3420'/><br> <target chassis='2' port='0x11'/><br> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/><br> </controller></div><div><br></div><div>~~~~~ snip ~~~~~~</div><div><br></div><div> <hostdev mode='subsystem' type='pci' managed='yes'><br> <source><br> <address domain='0x0000' bus='0xaf' slot='0x00' function='0x0'/><br> </source><br> <rom bar='on'/><br> <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0' multifunction='on'/><br> </hostdev><br> <hostdev mode='subsystem' type='pci' managed='yes'><br> <source><br> <address domain='0x0000' bus='0xaf' slot='0x00' function='0x1'/><br> </source><br> <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x1'/><br> </hostdev><br></div><div dir="ltr"><br></div><div>"model name=ioh3420" is Intel emulated pcie-root-port.</div><div>defalut "model name=pcie-root-port" is QEMU Paravirtualized pcie-root-port.</div><div>Radeon driver with "ioh3420" root port trick works fine on my workstation.</div><div>(but... not RX Vega 56, I'm using RX 460.)</div><div><br></div><div>Cheers,</div><div>Hisato.</div><div><br></div><div dir="ltr"><div dir="ltr"><br></div><div dir="ltr">> On Thu, 28 Mar 2019 at 20:32, Tmplt <tmplt dragons rocks> wrote:<br>> ><br>> > On 3/28/19 8:02 PM, Kash Pande wrote:<br>> > > kvm module parameter ignore_msrs is needed here.<br>> > Is this much different from the<br>> ><br>> > echo 1 > /sys/module/kvm/parameters/ignore_msrs<br>> <br>> I have a very similar setup with a Sapphire RX Vega 56 and also<br>> observing this lockup.<br>> I am using an unpatched Debian 4.19.0. Qemu 3.1.0. Libvirt 5.0.0. A<br>> single host core is pegged at 100%. I've attached my XML too.<br>> $ cat /sys/module/kvm/parameters/ignore_msrs<br>> Y<br>> I cannot find any logs indicating any problems...<br>> $ sudo journalctl | grep kvm<br>> Mar 28 20:35:52 socrates kernel: Command line:<br>> BOOT_IMAGE=/boot/vmlinuz-4.19.0-4-amd64<br>> root=/dev/mapper/VolGroup00-lvdebian ro iommu=pt<br>> amd_iommu=force_isolation rd.driver.pre=vfio-pci hugepagesz=1GB<br>> hugepages=16 vfio_iommu_type1.allow_unsafe_interrupts=1<br>> kvm.ignore_msrs=1<br>> Mar 28 20:35:52 socrates kernel: Kernel command line:<br>> BOOT_IMAGE=/boot/vmlinuz-4.19.0-4-amd64<br>> root=/dev/mapper/VolGroup00-lvdebian ro iommu=pt<br>> amd_iommu=force_isolation rd.driver.pre=vfio-pci hugepagesz=1GB<br>> hugepages=16 vfio_iommu_type1.allow_unsafe_interrupts=1<br>> kvm.ignore_msrs=1<br>> Mar 28 20:35:52 socrates kernel: kvm: Nested Virtualization enabled<br>> Mar 28 20:35:52 socrates kernel: kvm: Nested Paging enabled<br>> $ sudo journalctl | grep msrs<br>> Mar 28 20:35:52 socrates kernel: Command line:<br>> BOOT_IMAGE=/boot/vmlinuz-4.19.0-4-amd64<br>> root=/dev/mapper/VolGroup00-lvdebian ro iommu=pt<br>> amd_iommu=force_isolation rd.driver.pre=vfio-pci hugepagesz=1GB<br>> hugepages=16 vfio_iommu_type1.allow_unsafe_interrupts=1<br>> kvm.ignore_msrs=1<br>> Mar 28 20:35:52 socrates kernel: Kernel command line:<br>> BOOT_IMAGE=/boot/vmlinuz-4.19.0-4-amd64<br>> root=/dev/mapper/VolGroup00-lvdebian ro iommu=pt<br>> amd_iommu=force_isolation rd.driver.pre=vfio-pci hugepagesz=1GB<br>> hugepages=16 vfio_iommu_type1.allow_unsafe_interrupts=1<br>> kvm.ignore_msrs=1<br>> <br>> <domain type='kvm' xmlns:qemu='<a href="http://libvirt.org/schemas/domain/qemu/1.0">http://libvirt.org/schemas/domain/qemu/1.0</a>'><br>> <name>win10_backup</name><br>> <uuid>4b697e59-793e-450f-8722-38e785831796</uuid><br>> <title>Windows 10</title><br>> <description>Windows gaming VM</description><br>> <memory unit='KiB'>16777216</memory><br>> <currentMemory unit='KiB'>16777216</currentMemory><br>> <vcpu placement='static'>8</vcpu><br>> <iothreads>1</iothreads><br>> <cputune><br>> <vcpupin vcpu='0' cpuset='8'/><br>> <vcpupin vcpu='1' cpuset='9'/><br>> <vcpupin vcpu='2' cpuset='10'/><br>> <vcpupin vcpu='3' cpuset='11'/><br>> <vcpupin vcpu='4' cpuset='12'/><br>> <vcpupin vcpu='5' cpuset='13'/><br>> <vcpupin vcpu='6' cpuset='14'/><br>> <vcpupin vcpu='7' cpuset='15'/><br>> <emulatorpin cpuset='0-1'/><br>> <iothreadpin iothread='1' cpuset='0-1'/><br>> </cputune><br>> <os><br>> <type arch='x86_64' machine='pc-i440fx-3.1'>hvm</type><br>> <loader readonly='yes' type='pflash'>/usr/share/OVMF/OVMF_CODE.fd</loader><br>> <nvram>/var/lib/libvirt/qemu/nvram/win10_backup_VARS.fd</nvram><br>> </os><br>> <features><br>> <acpi/><br>> <apic/><br>> <hyperv><br>> <relaxed state='on'/><br>> <vapic state='on'/><br>> <spinlocks state='on' retries='8191'/><br>> </hyperv><br>> <vmport state='off'/><br>> </features><br>> <cpu mode='host-passthrough' check='none'><br>> <topology sockets='1' cores='4' threads='2'/><br>> <cache level='3' mode='emulate'/><br>> <feature policy='require' name='topoext'/><br>> </cpu><br>> <clock offset='localtime'><br>> <timer name='rtc' tickpolicy='catchup'/><br>> <timer name='pit' tickpolicy='delay'/><br>> <timer name='hpet' present='no'/><br>> <timer name='hypervclock' present='yes'/><br>> </clock><br>> <on_poweroff>destroy</on_poweroff><br>> <on_reboot>restart</on_reboot><br>> <on_crash>destroy</on_crash><br>> <pm><br>> <suspend-to-mem enabled='no'/><br>> <suspend-to-disk enabled='no'/><br>> </pm><br>> <devices><br>> <emulator>/usr/bin/kvm</emulator><br>> <disk type='file' device='disk'><br>> <driver name='qemu' type='raw' io='threads'/><br>> <source file='/var/lib/libvirt/images/win10-clean_backup.raw'/><br>> <target dev='vda' bus='virtio'/><br>> <boot order='2'/><br>> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/><br>> </disk><br>> <disk type='file' device='cdrom'><br>> <driver name='qemu' type='raw'/><br>> <source file='/var/lib/libvirt/images/virtio-win-0.1.160.iso'/><br>> <target dev='sda' bus='sata'/><br>> <readonly/><br>> <address type='drive' controller='0' bus='0' target='0' unit='0'/><br>> </disk><br>> <controller type='pci' index='0' model='pci-root'/><br>> <controller type='usb' index='0' model='nec-xhci'><br>> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/><br>> </controller><br>> <controller type='sata' index='0'><br>> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/><br>> </controller><br>> <interface type='network'><br>> <mac address='52:54:00:39:d1:fc'/><br>> <source network='default'/><br>> <model type='virtio'/><br>> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/><br>> </interface><br>> <input type='mouse' bus='ps2'/><br>> <input type='keyboard' bus='ps2'/><br>> <sound model='ich6'><br>> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/><br>> </sound><br>> <hostdev mode='subsystem' type='pci' managed='yes'><br>> <source><br>> <address domain='0x0000' bus='0x13' slot='0x00' function='0x0'/><br>> </source><br>> <rom bar='off'/><br>> <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/><br>> </hostdev><br>> <hostdev mode='subsystem' type='usb' managed='yes'><br>> <source><br>> <vendor id='0x046d'/><br>> <product id='0xc24c'/><br>> <address bus='3' device='7'/><br>> </source><br>> <address type='usb' bus='0' port='1'/><br>> </hostdev><br>> <hostdev mode='subsystem' type='usb' managed='yes'><br>> <source><br>> <vendor id='0x046d'/><br>> <product id='0xc228'/><br>> <address bus='3' device='9'/><br>> </source><br>> <address type='usb' bus='0' port='2'/><br>> </hostdev><br>> <memballoon model='virtio'><br>> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/><br>> </memballoon><br>> </devices><br>> <qemu:commandline><br>> <qemu:env name='QEMU_AUDIO_DRV' value='pa'/><br>> <qemu:env name='QEMU_PA_SERVER' value='/run/user/1000/pulse/native'/><br>> </qemu:commandline><br>> </domain><br><br></div></div></div></div></div></div>