[vfio-users] Ryzen Primary GPU passthrough success and woes
globalgorrilla at fastmail.fm
globalgorrilla at fastmail.fm
Mon Apr 3 17:13:41 UTC 2017
<memballoon model='none'/>
Might perform better.
Also, yes Hugepages might be helpful too.
Also, would you mind observing whether interrupts are posted when the
GPU is in use in pass-through?
watch -d cat /proc/interrupts
Look at the bottom for PIN (Posted-interrupt notification event) and PIW
(Posted-interrupt wakeup event).
Thanks!
On 30 Mar 2017, at 15:46, Graham Neville wrote:
> Finally gotten to the bottom of this, this the help of your XML file
> I'm
> able to run with a Q35 setup, I'm also able to just passthrough the
> CPU
> features. Thanks.
>
> I discovered the issue with the guest crashing was actually due to my
> graphics card overheating! Noticed that it was hitting 99C and then
> guest
> would crash.
>
> The setup is pretty sweet now, I just want to change my drives from
> sata to
> virtio which I believe will be quicker. Then also investigate
> HugePages
>
> Here's my Kernel params:
>
> cat /proc/cmdline
> BOOT_IMAGE=/vmlinuz-linux
> root=UUID=bf69add2-e36f-453a-b92e-a4343ca20d26 rw
> quiet amd_iommu=on vfio-pci.ids=1002:67b1,1002:aac8 video=efifb:off
> isolcpus=0-7
>
> Here's my final XML file:
>
> <domain type='kvm'>
> <name>Windows10</name>
> <uuid>d45c3b5f-be8a-41e8-a22c-02e91c4c6245</uuid>
> <memory unit='KiB'>8388608</memory>
> <currentMemory unit='KiB'>8388608</currentMemory>
> <vcpu placement='static'>4</vcpu>
> <cputune>
> <vcpupin vcpu='0' cpuset='0'/>
> <vcpupin vcpu='1' cpuset='1'/>
> <vcpupin vcpu='2' cpuset='2'/>
> <vcpupin vcpu='3' cpuset='3'/>
> </cputune>
> <os>
> <type arch='x86_64' machine='pc-q35-2.6'>hvm</type>
> <loader readonly='yes'
> type='pflash'>/home/virtualguests/windows10/ovmf_code_x64.bin</loader>
> <nvram>/home/virtualguests/windows10/ovmf_vars_x64.bin</nvram>
> <boot dev='hd'/>
> </os>
> <features>
> <acpi/>
> <apic/>
> <pae/>
> <hyperv>
> <relaxed state='on'/>
> <vapic state='on'/>
> <spinlocks state='on' retries='8191'/>
> </hyperv>
> <kvm>
> <hidden state='on'/>
> </kvm>
> </features>
> <cpu mode='host-passthrough'/>
> <clock offset='localtime'>
> <timer name='rtc' tickpolicy='catchup'/>
> <timer name='pit' tickpolicy='delay'/>
> <timer name='hpet' present='no'/>
> <timer name='hypervclock' present='yes'/>
> </clock>
> <on_poweroff>destroy</on_poweroff>
> <on_reboot>restart</on_reboot>
> <on_crash>restart</on_crash>
> <pm>
> <suspend-to-mem enabled='no'/>
> <suspend-to-disk enabled='no'/>
> </pm>
> <devices>
> <emulator>/usr/bin/qemu-system-x86_64</emulator>
> <disk type='file' device='disk'>
> <driver name='qemu' type='qcow2'/>
> <source
> file='/home/virtualguests/windows10/windows10-c-nas.qcow2'/>
> <target dev='sda' bus='sata'/>
> <address type='drive' controller='0' bus='0' target='0'
> unit='0'/>
> </disk>
> <disk type='file' device='disk'>
> <driver name='qemu' type='qcow2'/>
> <source file='/storage/windows10-d.qcow2'/>
> <target dev='sdb' bus='sata'/>
> <address type='drive' controller='0' bus='0' target='0'
> unit='1'/>
> </disk>
> <controller type='usb' index='0' model='ich9-ehci1'>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
> function='0x7'/>
> </controller>
> <controller type='usb' index='0' model='ich9-uhci1'>
> <master startport='0'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
> function='0x0' multifunction='on'/>
> </controller>
> <controller type='usb' index='0' model='ich9-uhci2'>
> <master startport='2'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
> function='0x1'/>
> </controller>
> <controller type='usb' index='0' model='ich9-uhci3'>
> <master startport='4'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
> function='0x2'/>
> </controller>
> <controller type='sata' index='0'>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f'
> function='0x2'/>
> </controller>
> <controller type='pci' index='0' model='pcie-root'/>
> <controller type='pci' index='1' model='dmi-to-pci-bridge'>
> <model name='i82801b11-bridge'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x1e'
> function='0x0'/>
> </controller>
> <controller type='pci' index='2' model='pci-bridge'>
> <model name='pci-bridge'/>
> <target chassisNr='2'/>
> <address type='pci' domain='0x0000' bus='0x01' slot='0x00'
> function='0x0'/>
> </controller>
> <controller type='pci' index='3' model='pcie-root-port'>
> <model name='ioh3420'/>
> <target chassis='3' port='0x40'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x08'
> function='0x0'/>
> </controller>
> <controller type='virtio-serial' index='0'>
> <address type='pci' domain='0x0000' bus='0x02' slot='0x03'
> function='0x0'/>
> </controller>
> <serial type='pty'>
> <target port='0'/>
> </serial>
> <console type='pty'>
> <target type='serial' port='0'/>
> </console>
> <input type='tablet' bus='usb'>
> <address type='usb' bus='0' port='1'/>
> </input>
> <input type='mouse' bus='ps2'/>
> <input type='keyboard' bus='ps2'/>
> <interface type='bridge'>
> <mac address='52:54:00:12:34:76'/>
> <source bridge='br0'/>
> <target dev='tap8'/>
> <model type='virtio'/>
> <alias name='virtio'/>
> <rom bar='off'/>
> </interface>
> <hostdev mode='subsystem' type='usb' managed='no'>
> <source>
> <vendor id='0x046d'/>
> <product id='0xc52e'/>
> </source>
> </hostdev>
> <hostdev mode='subsystem' type='usb' managed='no'>
> <source>
> <vendor id='0x04d8'/>
> <product id='0x00df'/>
> </source>
> </hostdev>
> <hostdev mode='subsystem' type='usb' managed='no'>
> <source>
> <vendor id='0x28de'/>
> <product id='0x1142'/>
> </source>
> </hostdev>
> <hostdev mode='subsystem' type='usb' managed='no'>
> <source>
> <vendor id='0x0a12'/>
> <product id='0x0001'/>
> </source>
> </hostdev>
> <hostdev mode='subsystem' type='usb' managed='no'>
> <source>
> <vendor id='0x0fcf'/>
> <product id='0x1009'/>
> </source>
> </hostdev>
> <hostdev mode='subsystem' type='usb' managed='no'>
> <source>
> <vendor id='0x1b1c'/>
> <product id='0x1c0b'/>
> </source>
> </hostdev>
> <sound model='ich6'>
> <address type='pci' domain='0x0000' bus='0x02' slot='0x02'
> function='0x0'/>
> </sound>
> <hostdev mode='subsystem' type='pci' managed='yes'>
> <source>
> <address domain='0x0000' bus='0x09' slot='0x00'
> function='0x0'/>
> </source>
> <rom bar='on' file='/home/virtualguests/windows10/r9290.rom'/>
> <address type='pci' domain='0x0000' bus='0x03' slot='0x00'
> function='0x0' multifunction='on'/>
> </hostdev>
> <hostdev mode='subsystem' type='pci' managed='yes'>
> <source>
> <address domain='0x0000' bus='0x09' slot='0x00'
> function='0x1'/>
> </source>
> <address type='pci' domain='0x0000' bus='0x03' slot='0x00'
> function='0x1'/>
> </hostdev>
> <memballoon model='virtio'>
> <address type='pci' domain='0x0000' bus='0x02' slot='0x04'
> function='0x0'/>
> </memballoon>
> </devices>
> </domain>
>
>
>
> On Thu, Mar 30, 2017 at 6:04 AM, Graham Neville
> <grahamneville at gmail.com>
> wrote:
>
>> I am using a ROM file which I dumped myself using GPU-Z, this was the
>> only
>> was I could get a display as well. With Q35 I do see TianoCore and
>> then the
>> Windows10 logo but the guest crashes straight after that. I may have
>> to try
>> and do a fresh install with Q35 to see if I can get further. Thanks
>> for the
>> XML, I'll try again tonight and report back.
>>
>>
>> On 29 Mar 2017 11:33 p.m., "Steven Walter" <stevenrwalter at gmail.com>
>> wrote:
>>
>> I didn't have good luck with Windows on BIOS or with Windows on i440.
>> I pretty much had to use Q35 + OVMF. You said you had trouble
>> getting
>> Windows to boot with Q35 + OVMF, did you at least get the TianoCore
>> logo to come up on your screen? If not you may need to dump and
>> specify the ROM file for your video card. For me OVMF would hang
>> unless I did this. Note that I'm using Q35 + OVMF, Opteron_G3 as
>> CPU,
>> specifying a ROM for the GPU, and running kvm_amd.avic=1.
>>
>> Here is my full guest XML file:
>>
>> <domain type='kvm'>
>> <name>WindowsTest</name>
>> <uuid>d45c3b5f-be8a-41e8-a22c-02e91c4c6245</uuid>
>> <memory unit='KiB'>8388608</memory>
>> <currentMemory unit='KiB'>8388608</currentMemory>
>> <vcpu placement='static'>8</vcpu>
>> <os>
>> <type arch='x86_64' machine='pc-q35-2.6'>hvm</type>
>> <loader readonly='yes' type='pflash'>/usr/share/OVMF/
>> OVMF_CODE.fd</loader>
>> <nvram>/var/lib/libvirt/qemu/nvram/WindowsTest_VARS.fd</nvram>
>> <boot dev='hd'/>
>> </os>
>> <features>
>> <acpi/>
>> <apic/>
>> <hyperv>
>> <relaxed state='on'/>
>> <vapic state='on'/>
>> <spinlocks state='on' retries='8191'/>
>> </hyperv>
>> </features>
>> <cpu mode='host-passthrough'/>
>> <clock offset='localtime'>
>> <timer name='rtc' tickpolicy='catchup'/>
>> <timer name='pit' tickpolicy='delay'/>
>> <timer name='hpet' present='no'/>
>> <timer name='hypervclock' present='yes'/>
>> </clock>
>> <on_poweroff>destroy</on_poweroff>
>> <on_reboot>restart</on_reboot>
>> <on_crash>restart</on_crash>
>> <pm>
>> <suspend-to-mem enabled='no'/>
>> <suspend-to-disk enabled='no'/>
>> </pm>
>> <devices>
>> <emulator>/usr/bin/kvm-spice</emulator>
>> <disk type='file' device='disk'>
>> <driver name='qemu' type='qcow2'/>
>> <source file='/var/lib/libvirt/images/WindowsTest.qcow2'/>
>> <target dev='sda' bus='sata'/>
>> <address type='drive' controller='0' bus='0' target='0'
>> unit='0'/>
>> </disk>
>> <disk type='file' device='cdrom'>
>> <driver name='qemu' type='raw'/>
>> <source file='/home/srwalter/Win10_1607_English_x64.iso'/>
>> <target dev='sdb' bus='sata'/>
>> <readonly/>
>> <address type='drive' controller='0' bus='0' target='0'
>> unit='1'/>
>> </disk>
>> <controller type='usb' index='0' model='ich9-ehci1'>
>> <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
>> function='0x7'/>
>> </controller>
>> <controller type='usb' index='0' model='ich9-uhci1'>
>> <master startport='0'/>
>> <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
>> function='0x0' multifunction='on'/>
>> </controller>
>> <controller type='usb' index='0' model='ich9-uhci2'>
>> <master startport='2'/>
>> <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
>> function='0x1'/>
>> </controller>
>> <controller type='usb' index='0' model='ich9-uhci3'>
>> <master startport='4'/>
>> <address type='pci' domain='0x0000' bus='0x00' slot='0x1d'
>> function='0x2'/>
>> </controller>
>> <controller type='sata' index='0'>
>> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f'
>> function='0x2'/>
>> </controller>
>> <controller type='pci' index='0' model='pcie-root'/>
>> <controller type='pci' index='1' model='dmi-to-pci-bridge'>
>> <model name='i82801b11-bridge'/>
>> <address type='pci' domain='0x0000' bus='0x00' slot='0x1e'
>> function='0x0'/>
>> </controller>
>> <controller type='pci' index='2' model='pci-bridge'>
>> <model name='pci-bridge'/>
>> <target chassisNr='2'/>
>> <address type='pci' domain='0x0000' bus='0x01' slot='0x00'
>> function='0x0'/>
>> </controller>
>> <controller type='pci' index='3' model='pcie-root-port'>
>> <model name='ioh3420'/>
>> <target chassis='3' port='0x40'/>
>> <address type='pci' domain='0x0000' bus='0x00' slot='0x08'
>> function='0x0'/>
>> </controller>
>> <controller type='virtio-serial' index='0'>
>> <address type='pci' domain='0x0000' bus='0x02' slot='0x03'
>> function='0x0'/>
>> </controller>
>> <serial type='pty'>
>> <target port='0'/>
>> </serial>
>> <console type='pty'>
>> <target type='serial' port='0'/>
>> </console>
>> <channel type='spicevmc'>
>> <target type='virtio' name='com.redhat.spice.0'/>
>> <address type='virtio-serial' controller='0' bus='0' port='1'/>
>> </channel>
>> <input type='tablet' bus='usb'>
>> <address type='usb' bus='0' port='1'/>
>> </input>
>> <input type='mouse' bus='ps2'/>
>> <input type='keyboard' bus='ps2'/>
>> <graphics type='spice' autoport='yes'>
>> <listen type='address'/>
>> </graphics>
>> <sound model='ich6'>
>> <address type='pci' domain='0x0000' bus='0x02' slot='0x02'
>> function='0x0'/>
>> </sound>
>> <video>
>> <model type='cirrus' vram='16384' heads='1' primary='yes'/>
>> <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
>> function='0x0'/>
>> </video>
>> <hostdev mode='subsystem' type='pci' managed='yes'>
>> <source>
>> <address domain='0x0000' bus='0x23' slot='0x00'
>> function='0x0'/>
>> </source>
>> <rom bar='on' file='/usr/share/OVMF/Ellesmere.rom'/>
>> <address type='pci' domain='0x0000' bus='0x03' slot='0x00'
>> function='0x0' multifunction='on'/>
>> </hostdev>
>> <hostdev mode='subsystem' type='pci' managed='yes'>
>> <source>
>> <address domain='0x0000' bus='0x23' slot='0x00'
>> function='0x1'/>
>> </source>
>> <address type='pci' domain='0x0000' bus='0x03' slot='0x00'
>> function='0x1'/>
>> </hostdev>
>> <redirdev bus='usb' type='spicevmc'>
>> <address type='usb' bus='0' port='2'/>
>> </redirdev>
>> <redirdev bus='usb' type='spicevmc'>
>> <address type='usb' bus='0' port='3'/>
>> </redirdev>
>> <memballoon model='virtio'>
>> <address type='pci' domain='0x0000' bus='0x02' slot='0x04'
>> function='0x0'/>
>> </memballoon>
>> </devices>
>> </domain>
>>
>> On Wed, Mar 29, 2017 at 5:44 PM, Graham Neville
>> <grahamneville at gmail.com>
>> wrote:
>>> Thanks for the link. I've tried a number of things but still no
>>> further
>> down
>>> the line.
>>>
>>> I've tried the following one-by-one
>>>
>>> kvm_amd.avic=1
>>> Properly isolated CPUs on host so they are exclusive to guest -
>> isolcpus=0-7
>>> Changed CPU host-passthrough to Opteron_G5 instead
>>> Changed CPU to althon
>>> Changed CPU to qemu64
>>> kvm-amd.npt=0
>>> iommu=pt
>>> Disabled SMT in BIOS
>>>
>>> In using Arch Linux with Kernel 4.10.1.
>>>
>>> uname -a
>>> Linux amdr7 4.10.5-1-ARCH #1 SMP PREEMPT Wed Mar 22 14:42:03 CET
>>> 2017
>> x86_64
>>> GNU/Linux
>>>
>>> This is my kernel command line now:
>>>
>>> BOOT_IMAGE=/vmlinuz-linux
>>> root=UUID=bf69add2-e36f-453a-b92e-a4343ca20d26
>> rw
>>> quiet amd_iommu=on vfio-pci.ids=1002:67b1,1002:aac8 video=efifb:off
>>> amdgpu.msi=0 kvm_amd.avic=1 isolcpus=0-7 kvm-amd.npt=0 iommu=pt
>>>
>>> This is my full libvirt XML file for the VM:
>>>
>>>
>>> <domain type='kvm'>
>>> <name>windows10</name>
>>> <uuid>7b222825-fc7d-4a66-a72c-5876063752d5</uuid>
>>> <memory unit='KiB'>8291456</memory>
>>> <currentMemory unit='KiB'>8291456</currentMemory>
>>> <vcpu placement='static'>4</vcpu>
>>> <cputune>
>>> <vcpupin vcpu='0' cpuset='0'/>
>>> <vcpupin vcpu='1' cpuset='1'/>
>>> <vcpupin vcpu='2' cpuset='2'/>
>>> <vcpupin vcpu='3' cpuset='3'/>
>>> </cputune>
>>> <os>
>>> <type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type>
>>> <loader type='pflash'
>>> readonly='yes'>/home/virtualguests/windows10/OVMF_CODE.fd</loader>
>>> <nvram>/home/virtualguests/windows10/OVMF_VARS.fd</nvram>
>>> <boot dev='hd'/>
>>> </os>
>>> <features>
>>> <acpi/>
>>> <apic/>
>>> <pae/>
>>> <hyperv>
>>> <relaxed state='on'/>
>>> <vapic state='on'/>
>>> <spinlocks state='on' retries='8191'/>
>>> </hyperv>
>>> <kvm>
>>> <hidden state='on'/>
>>> </kvm>
>>> </features>
>>> <cpus>
>>> <arch name='x86'>
>>> <model name='kvm64'>
>>> <feature name='apic'/>
>>> <feature name='clflush'/>
>>> <feature name='cmov'/>
>>> <feature name='cx16'/>
>>> <feature name='cx8'/>
>>> <feature name='de'/>
>>> <feature name='fpu'/>
>>> <feature name='fxsr'/>
>>> <feature name='lm'/>
>>> <feature name='mca'/>
>>> <feature name='mce'/>
>>> <feature name='mmx'/>
>>> <feature name='msr'/>
>>> <feature name='mtrr'/>
>>> <feature name='nx'/>
>>> <feature name='pae'/>
>>> <feature name='pat'/>
>>> <feature name='pge'/>
>>> <feature name='pni'/>
>>> <feature name='pse'/>
>>> <feature name='pse36'/>
>>> <feature name='sep'/>
>>> <feature name='sse'/>
>>> <feature name='sse2'/>
>>> <feature name='syscall'/>
>>> <feature name='tsc'/>
>>> </model>
>>> </arch>
>>> </cpus>
>>> <clock offset='localtime'>
>>> <timer name='rtc' tickpolicy='catchup'/>
>>> <timer name='pit' tickpolicy='delay'/>
>>> <timer name='hpet' present='no'/>
>>> <timer name='hypervclock' present='yes'/>
>>> </clock>
>>> <on_poweroff>destroy</on_poweroff>
>>> <on_reboot>restart</on_reboot>
>>> <on_crash>destroy</on_crash>
>>> <devices>
>>> <emulator>/usr/bin/qemu-system-x86_64</emulator>
>>> <disk type='file' device='disk'>
>>> <driver name='qemu' type='qcow2' />
>>> <source file='/home/virtualguests/wind
>> ows10/windows10-c-nas.qcow2'/>
>>> <target dev='vdb' bus='virtio'/>
>>> <address type='pci' domain='0x0000' bus='0x00' slot='0x07'
>>> function='0x0'/>
>>> </disk>
>>> <disk type='file' device='cdrom'>
>>> <driver name='qemu' type='raw'/>
>>> <source
>>> file='/home/virtualguests/windows10/Win10_1607_EnglishIntern
>> ational_x64.iso'/>
>>> <target dev='hdc' bus='ide'/>
>>> <readonly/>
>>> <address type='drive' controller='0' bus='1' unit='0'/>
>>> </disk>
>>> <disk type='file' device='disk'>
>>> <driver name='qemu' type='qcow2' cache='none'/>
>>> <source file='/storage/windows10-d.qcow2'/>
>>> <target dev='vdc' bus='virtio'/>
>>> </disk>
>>> <controller type='pci' index='0' model='pci-root' />
>>> <interface type='bridge'>
>>> <mac address='52:54:00:12:34:76'/>
>>> <source bridge='br0'/>
>>> <target dev='tap8'/>
>>> <model type='virtio'/>
>>> <alias name='virtio'/>
>>> <rom bar='off'/>
>>> </interface>
>>> <input type='mouse' bus='ps2'/>
>>> <input type='keyboard' bus='ps2'/>
>>> <hostdev mode='subsystem' type='usb' managed='no'>
>>> <source>
>>> <vendor id='0x046d'/>
>>> <product id='0xc52e'/>
>>> </source>
>>> </hostdev>
>>> <hostdev mode='subsystem' type='usb' managed='no'>
>>> <source>
>>> <vendor id='0x28de'/>
>>> <product id='0x1142'/>
>>> </source>
>>> </hostdev>
>>> <hostdev mode='subsystem' type='usb' managed='no'>
>>> <source>
>>> <vendor id='0x0a12'/>
>>> <product id='0x0001'/>
>>> </source>
>>> </hostdev>
>>> <hostdev mode='subsystem' type='usb' managed='no'>
>>> <source>
>>> <vendor id='0x0fcf'/>
>>> <product id='0x1009'/>
>>> </source>
>>> </hostdev>
>>> <hostdev mode='subsystem' type='usb' managed='no'>
>>> <source>
>>> <vendor id='0x1b1c'/>
>>> <product id='0x1c0b'/>
>>> </source>
>>> </hostdev>
>>> <hostdev mode='subsystem' type='usb' managed='no'>
>>> <source>
>>> <vendor id='0x05e3'/>
>>> <product id='0x0608'/>
>>> <address bus='1' device='5'/>
>>> </source>
>>> </hostdev>
>>> <hostdev mode='subsystem' type='pci' managed='yes'>
>>> <source>
>>> <address domain='0x0000' bus='0x09' slot='0x00'
>>> function='0x0' />
>>> </source>
>>> <rom bar='on' file='/home/virtualguests/windows10/r9290.rom'/>
>>> <address type='pci' domain='0x0000' bus='0x00' slot='0x08'
>>> function='0x0' multifunction='on'/>
>>> </hostdev>
>>> <hostdev mode='subsystem' type='pci' managed='yes'>
>>> <source>
>>> <address domain='0x0000' bus='0x09' slot='0x00'
>>> function='0x1'/>
>>> </source>
>>> <address type='pci' domain='0x0000' bus='0x00' slot='0x09'
>>> function='0x0'/>
>>> </hostdev>
>>> <memballoon model='none'/>
>>> </devices>
>>> </domain>
>>>
>>>
>>>
>>> When I have host-passthrough, Opteron_G5, althon or qemu64 CPUs
>> configured I
>>> see a lot of these stack traces just appearing frequently and not
>>> just
>> when
>>> the guest crashes, I see nothing when the guest crashes
>>>
>>> [ 2848.156709] ------------[ cut here ]------------
>>> [ 2848.156719] WARNING: CPU: 0 PID: 1445 at arch/x86/kvm/svm.c:1484
>>> avic_vcpu_load+0x15a/0x180 [kvm_amd]
>>> [ 2848.156720] Modules linked in: vhost_net vhost macvtap macvlan
>>> tun
>> nfsv3
>>> rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache hid_logitech_hidpp
>>> usb_serial_simple cdc_acm usbserial hid_logitech_dj cfg80211 bridge
>>> stp
>> llc
>>> amdgpu sd_mod edac_mce_amd radeon edac_core kvm_amd kvm
>>> crct10dif_pclmul
>>> crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc ppdev ttm
>>> snd_hda_codec_realtek snd_hda_codec_generic drm_kms_helper
>>> aesni_intel
>> drm
>>> btusb snd_hda_intel nls_iso8859_1 aes_x86_64 btrtl crypto_simd
>>> nls_cp437
>>> btbcm glue_helper syscopyarea btintel sysfillrect snd_hda_codec vfat
>> r8169
>>> joydev sysimgblt fat fb_sys_fops i2c_algo_bit bluetooth cryptd evdev
>>> mousedev uas mii input_leds snd_hda_core rfkill pcspkr led_class
>> snd_hwdep
>>> mac_hid snd_pcm snd_timer ccp sp5100_tco snd i2c_piix4 soundcore
>>> rng_core
>>> shpchp wmi parport_pc
>>> [ 2848.156772] parport fjes 8250_dw i2c_designware_platform
>>> tpm_infineon
>>> i2c_designware_core button acpi_cpufreq tpm_tis tpm_tis_core tpm
>>> nfsd
>>> auth_rpcgss oid_registry nfs_acl lockd grace sch_fq_codel sunrpc
>> ip_tables
>>> x_tables ext4 crc16 jbd2 fscrypto mbcache usb_storage hid_generic
>>> usbhid
>> hid
>>> ahci libahci xhci_pci libata xhci_hcd usbcore scsi_mod nvme
>>> usb_common
>>> nvme_core serio vfio_pci irqbypass vfio_virqfd vfio_iommu_type1 vfio
>>> [ 2848.156798] CPU: 0 PID: 1445 Comm: CPU 0/KVM Tainted: G W
>>> 4.10.5-1-ARCH #1
>>> [ 2848.156798] Hardware name: Gigabyte Technology Co., Ltd. Default
>>> string/AB350M-Gaming 3-CF, BIOS F2 02/20/2017
>>> [ 2848.156799] Call Trace:
>>> [ 2848.156807] dump_stack+0x63/0x83
>>> [ 2848.156812] __warn+0xcb/0xf0
>>> [ 2848.156816] warn_slowpath_null+0x1d/0x20
>>> [ 2848.156819] avic_vcpu_load+0x15a/0x180 [kvm_amd]
>>> [ 2848.156822] svm_vcpu_unblocking+0x18/0x20 [kvm_amd]
>>> [ 2848.156834] kvm_vcpu_block+0xd3/0x330 [kvm]
>>> [ 2848.156844] ? kvm_get_rflags+0x1a/0x30 [kvm]
>>> [ 2848.156856] kvm_arch_vcpu_ioctl_run+0x4ea/0x1680 [kvm]
>>> [ 2848.156859] ? _copy_to_user+0x54/0x60
>>> [ 2848.156867] kvm_vcpu_ioctl+0x339/0x630 [kvm]
>>> [ 2848.156872] do_vfs_ioctl+0xa3/0x5f0
>>> [ 2848.156876] ? __fget+0x77/0xb0
>>> [ 2848.156880] SyS_ioctl+0x79/0x90
>>> [ 2848.156883] entry_SYSCALL_64_fastpath+0x1a/0xa9
>>> [ 2848.156885] RIP: 0033:0x7f9980dbd0d7
>>> [ 2848.156886] RSP: 002b:00007f9972efb8e8 EFLAGS: 00000246 ORIG_RAX:
>>> 0000000000000010
>>> [ 2848.156887] RAX: ffffffffffffffda RBX: 00007f9987e0d001 RCX:
>>> 00007f9980dbd0d7
>>> [ 2848.156888] RDX: 0000000000000000 RSI: 000000000000ae80 RDI:
>>> 0000000000000013
>>> [ 2848.156888] RBP: 0000000000000001 R08: 000055c65eff4830 R09:
>>> 00000000000000ff
>>> [ 2848.156889] R10: 0000000000000001 R11: 0000000000000246 R12:
>>> 0000000000000001
>>> [ 2848.156889] R13: 00007f9987e0c000 R14: 0000000000000000 R15:
>>> 00007f99745a5980
>>> [ 2848.156905] ---[ end trace e49522bc58864bce ]---
>>>
>>> I still haven't tried SeaBIOS yet, I'm still running 'pc-i440fx-2.1'
>>> with
>>> OVMF, not Q35. I couldn't get Windows to boot with Q35.
>>>
>>> I also noticed something really odd in the fact that after the guest
>> crashes
>>> I see random pictures on the TV, which I assume are coming from Arch
>>> - I
>> see
>>> things like a woman at a football stadium and waterfalls - I'm not
>>> sure
>> if
>>> this would be expected if the card is assigned to vfio-pci?
>>>
>>> @Steven Walter, can you paste a full copy of your libvirt XML file
>> please?
>>>
>>> Just for completeness here are my IOMMU groups:
>>>
>>> [gneville at amdr7 ~]$ lspci -nn
>>> 00:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD]
>>> Device
>>> [1022:1450]
>>> 00:00.2 IOMMU [0806]: Advanced Micro Devices, Inc. [AMD] Device
>> [1022:1451]
>>> 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD]
>>> Device
>>> [1022:1452]
>>> 00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device
>>> [1022:1453]
>>> 00:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device
>>> [1022:1453]
>>> 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD]
>>> Device
>>> [1022:1452]
>>> 00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD]
>>> Device
>>> [1022:1452]
>>> 00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device
>>> [1022:1453]
>>> 00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD]
>>> Device
>>> [1022:1452]
>>> 00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD]
>>> Device
>>> [1022:1452]
>>> 00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device
>>> [1022:1454]
>>> 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD]
>>> Device
>>> [1022:1452]
>>> 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device
>>> [1022:1454]
>>> 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus
>>> Controller [1022:790b] (rev 59)
>>> 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH
>>> LPC
>> Bridge
>>> [1022:790e] (rev 51)
>>> 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD]
>>> Device
>>> [1022:1460]
>>> 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD]
>>> Device
>>> [1022:1461]
>>> 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD]
>>> Device
>>> [1022:1462]
>>> 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD]
>>> Device
>>> [1022:1463]
>>> 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD]
>>> Device
>>> [1022:1464]
>>> 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD]
>>> Device
>>> [1022:1465]
>>> 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD]
>>> Device
>>> [1022:1466]
>>> 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD]
>>> Device
>>> [1022:1467]
>>> 01:00.0 Non-Volatile memory controller [0108]: Samsung Electronics
>>> Co Ltd
>>> Device [144d:a804]
>>> 03:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD]
>>> Device
>>> [1022:43bb] (rev 02)
>>> 03:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD]
>>> Device
>>> [1022:43b7] (rev 02)
>>> 03:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device
>>> [1022:43b2] (rev 02)
>>> 04:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device
>>> [1022:43b4] (rev 02)
>>> 04:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device
>>> [1022:43b4] (rev 02)
>>> 04:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device
>>> [1022:43b4] (rev 02)
>>> 05:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd.
>>> RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
>>> [10ec:8168]
>> (rev
>>> 0c)
>>> 07:00.0 Non-Volatile memory controller [0108]: Samsung Electronics
>>> Co Ltd
>>> Device [144d:a804]
>>> 09:00.0 VGA compatible controller [0300]: Advanced Micro Devices,
>>> Inc.
>>> [AMD/ATI] Hawaii PRO [Radeon R9 290/390] [1002:67b1]
>>> 09:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI]
>> Hawaii
>>> HDMI Audio [Radeon R9 290/290X / 390/390X] [1002:aac8]
>>> 11:00.0 Non-Essential Instrumentation [1300]: Advanced Micro
>>> Devices,
>> Inc.
>>> [AMD] Device [1022:145a]
>>> 11:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc.
>>> [AMD]
>>> Device [1022:1456]
>>> 11:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD]
>>> Device
>>> [1022:145c]
>>> 12:00.0 Non-Essential Instrumentation [1300]: Advanced Micro
>>> Devices,
>> Inc.
>>> [AMD] Device [1022:1455]
>>> 12:00.2 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD]
>>> FCH
>> SATA
>>> Controller [AHCI mode] [1022:7901] (rev 51)
>>> 12:00.3 Audio device [0403]: Advanced Micro Devices, Inc. [AMD]
>>> Device
>>> [1022:1457]
>>> [gneville at amdr7 ~]$
>>>
>>> virsh nodedev-dumpxml pci_0000_09_00_0
>>> <device>
>>> <name>pci_0000_09_00_0</name>
>>> <path>/sys/devices/pci0000:00/0000:00:03.1/0000:09:00.0</path>
>>> <parent>pci_0000_00_03_1</parent>
>>> <driver>
>>> <name>vfio-pci</name>
>>> </driver>
>>> <capability type='pci'>
>>> <domain>0</domain>
>>> <bus>9</bus>
>>> <slot>0</slot>
>>> <function>0</function>
>>> <product id='0x67b1'>Hawaii PRO [Radeon R9 290/390]</product>
>>> <vendor id='0x1002'>Advanced Micro Devices, Inc.
>>> [AMD/ATI]</vendor>
>>> <iommuGroup number='2'>
>>> <address domain='0x0000' bus='0x00' slot='0x03'
>>> function='0x0'/>
>>> <address domain='0x0000' bus='0x09' slot='0x00'
>>> function='0x1'/>
>>> <address domain='0x0000' bus='0x00' slot='0x03'
>>> function='0x1'/>
>>> <address domain='0x0000' bus='0x09' slot='0x00'
>>> function='0x0'/>
>>> </iommuGroup>
>>> </capability>
>>> </device>
>>>
>>> find /sys/kernel/iommu_groups/ -type l
>>> /sys/kernel/iommu_groups/7/devices/0000:00:18.6
>>> /sys/kernel/iommu_groups/7/devices/0000:00:18.4
>>> /sys/kernel/iommu_groups/7/devices/0000:00:18.2
>>> /sys/kernel/iommu_groups/7/devices/0000:00:18.0
>>> /sys/kernel/iommu_groups/7/devices/0000:00:18.7
>>> /sys/kernel/iommu_groups/7/devices/0000:00:18.5
>>> /sys/kernel/iommu_groups/7/devices/0000:00:18.3
>>> /sys/kernel/iommu_groups/7/devices/0000:00:18.1
>>> /sys/kernel/iommu_groups/5/devices/0000:12:00.2
>>> /sys/kernel/iommu_groups/5/devices/0000:00:08.1
>>> /sys/kernel/iommu_groups/5/devices/0000:12:00.0
>>> /sys/kernel/iommu_groups/5/devices/0000:12:00.3
>>> /sys/kernel/iommu_groups/5/devices/0000:00:08.0
>>> /sys/kernel/iommu_groups/3/devices/0000:00:04.0
>>> /sys/kernel/iommu_groups/1/devices/0000:00:02.0
>>> /sys/kernel/iommu_groups/6/devices/0000:00:14.0
>>> /sys/kernel/iommu_groups/6/devices/0000:00:14.3
>>> /sys/kernel/iommu_groups/4/devices/0000:11:00.2
>>> /sys/kernel/iommu_groups/4/devices/0000:11:00.0
>>> /sys/kernel/iommu_groups/4/devices/0000:00:07.1
>>> /sys/kernel/iommu_groups/4/devices/0000:11:00.3
>>> /sys/kernel/iommu_groups/4/devices/0000:00:07.0
>>> /sys/kernel/iommu_groups/2/devices/0000:00:03.0
>>> /sys/kernel/iommu_groups/2/devices/0000:09:00.1
>>> /sys/kernel/iommu_groups/2/devices/0000:00:03.1
>>> /sys/kernel/iommu_groups/2/devices/0000:09:00.0
>>> /sys/kernel/iommu_groups/0/devices/0000:07:00.0
>>> /sys/kernel/iommu_groups/0/devices/0000:03:00.1
>>> /sys/kernel/iommu_groups/0/devices/0000:00:01.3
>>> /sys/kernel/iommu_groups/0/devices/0000:04:01.0
>>> /sys/kernel/iommu_groups/0/devices/0000:00:01.1
>>> /sys/kernel/iommu_groups/0/devices/0000:04:04.0
>>> /sys/kernel/iommu_groups/0/devices/0000:05:00.0
>>> /sys/kernel/iommu_groups/0/devices/0000:04:00.0
>>> /sys/kernel/iommu_groups/0/devices/0000:03:00.2
>>> /sys/kernel/iommu_groups/0/devices/0000:03:00.0
>>> /sys/kernel/iommu_groups/0/devices/0000:00:01.0
>>> /sys/kernel/iommu_groups/0/devices/0000:01:00.0
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Wed, Mar 29, 2017 at 12:24 PM, Steven Walter
>>> <stevenrwalter at gmail.com
>>>
>>> wrote:
>>>>
>>>> I got a similar (though multi-GPU) setup working, which I wrote up
>>>> here:
>>>> https://www.reddit.com/r/VFIO/comments/616xih/gpu_passthroug
>> h_with_msi_b350_tomahawk/
>>>>
>>>> One thing that may help you is to enable AVIC (kvm_amd.avic=1).
>>>> What
>>>> I saw without AVIC was that things would work briefly (only a few
>>>> seconds for me) before interrupts would stop getting delivered.
>>>> Sounds like things are working better for you without AVIC than
>>>> they
>>>> did for me, but perhaps the extra improvement in IRQ latency would
>>>> fix
>>>> the hangs you get during intensive graphics operations.
>>>>
>>>>
>>>> On Tue, Mar 28, 2017 at 6:02 PM, Graham Neville <
>> grahamneville at gmail.com>
>>>> wrote:
>>>>> I've managed to get pci-e passthough working on a gigabyte gaming
>>>>> 3
>> matx
>>>>> MB
>>>>> and Ryzen 1700, no ACS patch, using only 1 GPU - AMD r9 290.
>>>>> However
>> I'm
>>>>> facing a problem with the whole KVM setup and not sure what it's
>> related
>>>>> to.
>>>>> For the Windows10 guest with the GPU passed through it crashes
>>>>> (guest
>>>>> only,
>>>>> host is fine) whenever I try anything graphics intensive, for
>>>>> example
>>>>> running Witcher3. Normal desktop is fine.
>>>>> Also my Linux guests are acting odd when I try to SSH to them, I
>> notice
>>>>> that
>>>>> the SSH terminals just stop working randomly. And then there's the
>> issue
>>>>> with very slow network throughout to both VMs. I have no idea
>>>>> what's
>>>>> going
>>>>> on. It used to work fine with my Intel setup. There's no logs in
>>>>> dmesg
>>>>> to
>>>>> show a problem either.
>>>>>
>>>>> I'm going to try Seabios instead of OVMF to see if I can stop the
>>>>> crashing.
>>>>>
>>>>> Any one having similar issues or anyone can advise?
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> vfio-users mailing list
>>>>> vfio-users at redhat.com
>>>>> https://www.redhat.com/mailman/listinfo/vfio-users
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> -Steven Walter <stevenrwalter at gmail.com>
>>>
>>>
>>
>>
>>
>> --
>> -Steven Walter <stevenrwalter at gmail.com>
>>
>>
>>
> _______________________________________________
> vfio-users mailing list
> vfio-users at redhat.com
> https://www.redhat.com/mailman/listinfo/vfio-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20170403/78fca946/attachment.htm>
More information about the vfio-users
mailing list