<div dir="ltr"><div>I have some persistent issues with a couple USB devices via vfio/qemu that I do not experience when booted native:</div><div> </div>Oculus Rift CV1 - head-tracking stutter (any game or demo)<div>- host CPU ~100% on every core (any game or demo)</div><div> - at same time Guest CPU may only show ~20% usage - audio static from built-in USB DAC</div><div> FiiO K1 USB Headphone DAC/AMP<div>- mild but noticeable cuts/jitter</div><div> </div><div>This may also impact other USB devices in ways I have not yet noticed. Any kind of test to quantify this would be helpful. Guest OS was installed on a dedicated disk so I can easily native boot to compare tests when needed.</div><div> </div><div>3DMark score is only 0.64% lower in VM which indicates to me at least things are -mostly- there: <a href="http://www.3dmark.com/compare/fs/8402818/fs/8403766">http://www.3dmark.com/compare/fs/8402818/fs/8403766</a></div><div><pre style="color:rgb(0,0,0)">Guest OS is Windows 10 Professional.</pre><pre style="color:rgb(0,0,0)">Any advice/debugging tips would be most welcome. </pre></div><div>Here are all the other relevant details I can think of for my setup:</div><div><pre style="color:rgb(0,0,0)">> uname -r 4.4.5-1-vfio > lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 8 On-line CPU(s) list: 0-7 Thread(s) per core: 2 Core(s) per socket: 4 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 94 Model name: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz Stepping: 3 CPU MHz: 4200.000 CPU max MHz: 4200.0000 CPU min MHz: 800.0000 BogoMIPS: 8019.82 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 8192K NUMA node0 CPU(s): 0-7 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch ida arat epb pln pts dtherm hwp hwp_notify hwp_act_window hwp_epp intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 > sudo lshw -short H/W path Device Class Description ================================================================= system System Product Name (SKU) /0 bus MAXIMUS VIII EXTREME /0/0 memory 64KiB BIOS /0/42 memory 128KiB L1 cache /0/43 memory 128KiB L1 cache /0/44 memory 1MiB L2 cache /0/45 memory 8MiB L3 cache /0/46 processor Core i7 (To Be Filled By O.E.M.) /0/47 memory 32GiB System Memory /0/47/0 memory 16GiB DIMM Synchronous 2133 MHz (0.5 ns) /0/47/1 memory [empty] /0/47/2 memory 16GiB DIMM Synchronous 2133 MHz (0.5 ns) /0/47/3 memory [empty] /0/100 bridge Skylake Host Bridge/DRAM Registers /0/100/1 bridge Skylake PCIe Controller (x16) /0/100/1/0 display GM200 [GeForce GTX 980 Ti] /0/100/1/0.1 multimedia NVIDIA Corporation /0/100/1.1 bridge Skylake PCIe Controller (x8) /0/100/1.1/0 display Caicos PRO [Radeon HD 7450] /0/100/1.1/0.1 multimedia Caicos HDMI Audio [Radeon HD 6400 Series] /0/100/2 display Skylake Integrated Graphics /0/100/14 bus Sunrise Point-H USB 3.0 xHCI Controller /0/100/16 communication Sunrise Point-H CSME HECI #1 /0/100/17 storage Sunrise Point-H SATA controller [AHCI mode] /0/100/1b bridge Sunrise Point-H PCI Root Port #17 /0/100/1c bridge Sunrise Point-H PCI Express Root Port #1 /0/100/1c/0 bus ASM1142 USB 3.1 Host Controller /0/100/1c/0/0 usb2 bus xHCI Host Controller /0/100/1c/0/0/1 storage CDDVDW SU-208DB /0/100/1c/0/1 usb1 bus xHCI Host Controller /0/100/1c/0/1/2 multimedia HD Pro Webcam C920 /0/100/1c.2 bridge Sunrise Point-H PCI Express Root Port #3 /0/100/1c.2/0 bridge ASMedia Technology Inc. /0/100/1c.2/0/1 bridge ASMedia Technology Inc. /0/100/1c.2/0/1/0 bus VL805 USB 3.0 Host Controller /0/100/1c.2/0/1/0/0 usb4 bus xHCI Host Controller /0/100/1c.2/0/1/0/1 usb3 bus xHCI Host Controller /0/100/1c.2/0/1/0/1/1 bus USB2.0 Hub /0/100/1c.2/0/1/0/1/1/4 generic FT240X USB FIFO /0/100/1c.2/0/2 bridge ASMedia Technology Inc. /0/100/1c.2/0/2/0 network BCM4360 802.11ac Wireless Network Adapter /0/100/1c.2/0/3 bridge ASMedia Technology Inc. /0/100/1c.2/0/3/0 storage ASM1062 Serial ATA Controller /0/100/1c.2/0/4 bridge ASMedia Technology Inc. /0/100/1c.2/0/5 bridge ASMedia Technology Inc. /0/100/1c.2/0/5/0 bus ASM1042A USB 3.0 Host Controller /0/100/1c.2/0/5/0/0 usb6 bus xHCI Host Controller /0/100/1c.2/0/5/0/0/2 bus USB3.0 Hub /0/100/1c.2/0/5/0/0/2/4 bus USB3.0 Hub /0/100/1c.2/0/5/0/1 usb5 bus xHCI Host Controller /0/100/1c.2/0/5/0/1/1 generic USB 10/100 LAN /0/100/1c.2/0/5/0/1/2 bus USB2.0 Hub /0/100/1c.2/0/5/0/1/2/1 bus USB2.0 Hub /0/100/1c.2/0/5/0/1/2/4 bus USB2.0 Hub /0/100/1c.2/0/5/0/1/2/4/4 input Yubikey 4 OTP+U2F+CCID /0/100/1c.2/0/6 bridge ASMedia Technology Inc. /0/100/1c.2/0/7 bridge ASMedia Technology Inc. /0/100/1c.4 bridge Sunrise Point-H PCI Express Root Port #5 /0/100/1c.4/0 bridge DSL6540 Thunderbolt 3 Bridge [Alpine Ridge 4C 2015] /0/100/1c.4/0/0 bridge DSL6540 Thunderbolt 3 Bridge [Alpine Ridge 4C 2015] /0/100/1c.4/0/1 bridge DSL6540 Thunderbolt 3 Bridge [Alpine Ridge 4C 2015] /0/100/1c.4/0/2 bridge DSL6540 Thunderbolt 3 Bridge [Alpine Ridge 4C 2015] /0/100/1c.4/0/2/0 bus DSL6540 USB 3.1 Controller [Alpine Ridge] /0/100/1c.4/0/2/0/0 usb8 bus xHCI Host Controller /0/100/1c.4/0/2/0/1 usb7 bus xHCI Host Controller /0/100/1c.4/0/2/0/1/2 bus USB 2.0 Hub /0/100/1c.4/0/2/0/1/2/2 input CST Laser Trackball /0/100/1c.4/0/2/0/1/2/3 input ErgoDox ergonomic keyboard /0/100/1c.4/0/4 bridge DSL6540 Thunderbolt 3 Bridge [Alpine Ridge 4C 2015] /0/100/1d bridge Sunrise Point-H PCI Express Root Port #9 /0/100/1d/0 storage NVMe SSD Controller /0/100/1f bridge Sunrise Point-H LPC Controller /0/100/1f.2 memory Memory controller /0/100/1f.3 multimedia Sunrise Point-H HD Audio /0/100/1f.4 bus Sunrise Point-H SMBus /0/100/1f.6 enp0s31f6 network Ethernet Connection (2) I219-V /1 power To Be Filled By O.E.M. /2 scsi8 storage /3 virbr0-nic network Ethernet interface /4 enp11s0u1 network Ethernet interface > cat /proc/cmdline initrd=\initramfs-linux-vfio.img cryptdevice=UUID=bc838115-107d-41ef-a076-6a59a2a6bb43:cryptroot root=/dev/mapper/cryptroot pcie_acs_override=downstream intel_iommu=on rw > dmesg | grep -e DMAR -e IOMMU [ 0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA [ 0.000000] ACPI: DMAR 0x00000000365D85A8 0000B8 (v01 INTEL SKL 00000001 INTL 00000001) [ 0.000000] DMAR: IOMMU enabled [ 0.076983] DMAR: Host address width 39 [ 0.076984] DMAR: DRHD base: 0x000000fed90000 flags: 0x0 [ 0.076989] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap 1c0000c40660462 ecap 7e3ff0505e [ 0.076990] DMAR: DRHD base: 0x000000fed91000 flags: 0x1 [ 0.076993] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da [ 0.076994] DMAR: RMRR base: 0x0000003630a000 end: 0x00000036329fff [ 0.076994] DMAR: RMRR base: 0x00000038800000 end: 0x0000003affffff [ 0.076996] DMAR-IR: IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1 [ 0.076996] DMAR-IR: HPET id 0 under DRHD base 0xfed91000 [ 0.076997] DMAR-IR: x2apic is disabled because BIOS sets x2apic opt out bit. [ 0.076997] DMAR-IR: Use 'intremap=no_x2apic_optout' to override the BIOS setting. [ 0.078318] DMAR-IR: Enabled IRQ remapping in xapic mode [ 0.587024] DMAR: [Firmware Bug]: RMRR entry for device 04:00.0 is broken - applying workaround [ 0.587027] DMAR: [Firmware Bug]: RMRR entry for device 0b:00.0 is broken - applying workaround [ 0.587030] DMAR: No ATSR found [ 0.587086] DMAR: dmar0: Using Queued invalidation [ 0.587296] DMAR: dmar1: Using Queued invalidation [ 0.587301] DMAR: Setting RMRR: [ 0.587327] DMAR: Setting identity map for device 0000:00:02.0 [0x38800000 - 0x3affffff] [ 0.587335] DMAR: Setting identity map for device 0000:00:14.0 [0x3630a000 - 0x36329fff] [ 0.587343] DMAR: Setting identity map for device 0000:04:00.0 [0x3630a000 - 0x36329fff] [ 0.587350] DMAR: Setting identity map for device 0000:0b:00.0 [0x3630a000 - 0x36329fff] [ 0.587353] DMAR: Prepare 0-16MiB unity mapping for LPC [ 0.587357] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff] [ 0.587383] DMAR: Intel(R) Virtualization Technology for Directed I/O [ 108.828150] AMD IOMMUv2 driver by Joerg Roedel <<a href="mailto:jroedel@suse.de">jroedel@suse.de</a>> [ 108.828152] AMD IOMMUv2 functionality not available on this system > for id in 00:14.0 01:00.0 01:00.1; do sudo lspci -nn -v -s $id; done 00:14.0 USB controller [0c03]: Intel Corporation Sunrise Point-H USB 3.0 xHCI Controller [8086:a12f] (rev 31) (prog-if 30 [XHCI]) Subsystem: ASUSTeK Computer Inc. Device [1043:8694] Flags: bus master, medium devsel, latency 0, IRQ 171 Memory at 2fff010000 (64-bit, non-prefetchable) [size=64K] Capabilities: [70] Power Management version 2 Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+ Kernel driver in use: vfio-pci Kernel modules: xhci_pci 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM200 [GeForce GTX 980 Ti] [10de:17c8] (rev a1) (prog-if 00 [VGA controller]) Subsystem: eVga.com. Corp. Device [3842:4995] Flags: bus master, fast devsel, latency 0, IRQ 170 Memory at db000000 (32-bit, non-prefetchable) [size=16M] Memory at 90000000 (64-bit, prefetchable) [size=256M] Memory at a0000000 (64-bit, prefetchable) [size=32M] I/O ports at e000 [size=128] Expansion ROM at dc000000 [disabled] [size=512K] Capabilities: [60] Power Management version 3 Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+ Capabilities: [78] Express Legacy Endpoint, MSI 00 Capabilities: [100] Virtual Channel Capabilities: [258] L1 PM Substates Capabilities: [128] Power Budgeting <?> Capabilities: [420] Advanced Error Reporting Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?> Capabilities: [900] #19 Kernel driver in use: vfio-pci Kernel modules: nouveau 01:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fb0] (rev a1) Subsystem: eVga.com. Corp. Device [3842:4995] Flags: bus master, fast devsel, latency 0, IRQ 17 Memory at dc080000 (32-bit, non-prefetchable) [size=16K] Capabilities: [60] Power Management version 3 Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [78] Express Endpoint, MSI 00 Capabilities: [100] Advanced Error Reporting Kernel driver in use: vfio-pci Kernel modules: snd_hda_intel > sudo sysctl vm.nr_hugepages=7168 vm.nr_hugepages = 7168 > sudo qemu-system-x86_64 \ -enable-kvm \ -machine type=q35,accel=kvm \ -cpu host,kvm=off \ -smp 8,sockets=1,cores=4,threads=2 \ -m 8G \ -mem-path /dev/hugepages \ -device virtio-scsi-pci,id=scsi \ -drive file=/dev/sda,id=disk,format=raw,if=none \ -device scsi-hd,drive=disk \ -device vfio-pci,host=01:00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1 \ -drive if=pflash,format=raw,readonly,file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd \ -drive if=pflash,format=raw,file=/tmp/OVMF_VARS-pure-efi.fd \ -nographic \ -vga none \ -usb \ -device vfio-pci,host=00:14.0 \ -rtc clock=host,base=utc</pre></div><div> </div>-- <div class="gmail_signature">Lance R. Vick </div> </div></div>