[vfio-users] I need your help!

Fri Feb 24 01:24:18 UTC 2017

Thank you for your response, :) 

And thank you for bringing isolcpus to my attention, I’m about to test it and configure the vCPU pinning with numactl / taskset. I will also try adding the hypervisor option to the host flags however I believe the default is to copy the host CPU configuration as much as possible. 

I still have a few questions including a new one about the vCPU process itself. 

Unfortunately I can’t run this in libvirt at the moment without figuring out the conversion tool to change the shell script into a domain XML file. IIRC I had problems converting it and decided it was too much hassle to get working, also has something to do with why I initially set the permissions to be releaxed on /dev/vfio in the script that I provided, it seemed like more trouble than it was worth at the time but I’m open to revisiting. 

Here’s the entire process list for the qemu process after it starts: https://gist.github.com/cloudkitsch/eefec59233269ace0c9707d0a57e0d5f#file-ps-elf-9999-L3

I think lines 3-9 are the vCPU threads however I have no way to tell definitively without looking at the CPU usage percentage (5th column.) I’m not really sure how libvirt does this, but if I had to take a guess I’d probably say numactl / taskset or related libraries are involved somehow? Trying to think of a way I could script this, but it seems difficult to imagine without additional process/thread labeling to tell the threads apart. Would you or anybody else be aware of a way? I looked up to see if there’s a -vcpu for qemu but it doesn’t exist yet?

The other question I had was about the default behavior of -machine, (tianocore’s BIOS seems to think that its something between i440fx, q35 and ICH9.) Also I don’t really understand why I can’t get q35 to work or if I even need to. Does this matter? It seems very complicated, but I’m not sure where to start as far as understanding why it’s complicated. Q35 does work just not with my video card as I lose video as soon as it starts the driver on Windows. There was a thread on here a while ago where someone was trying to understand this as well but I couldn’t make heads or tails over whether or not anyone ever arrived at a conclusion.

Thanks again, I’ll give isolcpus a try later tonight or tomorrow and update you. Feel free to update me between then and now about this stuff, too. 

-Paige

From: Bronek Kozicki [mailto:brok at spamcop.net] 
Sent: Thursday, February 23, 2017 4:03 AM
To: vfio-users at redhat.com; Paige Thompson <paigeadele at gmail.com>
Subject: Re: [vfio-users] I need your help!

Hi Paige

I noticed your -cpu option does not disable hypervisor; you might want to try it, please see http://vfio.blogspot.co.uk/2016/10/how-to-improve-performance-in-windows-7.html for reference. On the example of my virtual machines, this translates to qemu options:

-machine pc-i440fx-2.5,accel=kvm,usb=off,dump-guest-core=off,mem-merge=off -cpu host,-hypervisor,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1000

Another thing, I make use of vCPU pinning in my libvirt configuration and I think this would help you as well. The CPUs I use for virtual machines are removed from Linux kernel scheduler, with "isolcpus" option. FWIW my motherboard is SuperMicro X9DA7. And one more thing, when setting up my VMs I took particular care to ensure that the physical cores I pinned my vCPUs to, are all on the same socket and directly attached to PCIe slots that I mapped to my VMs with vfio.

B.

--

  Bronek Kozicki

  brok at spamcop.net <mailto:brok at spamcop.net> 

On Thu, 23 Feb 2017, at 11:22 AM, Paige Thompson wrote:

I have been running this setup since October and I have tried since this time to fix the same issues without any success either in terms of isolating the problem or fixing it. Here’s a lot of details/specs about the setup that I could think of off-hand to include:

 <https://gist.github.com/cloudkitsch/eefec59233269ace0c9707d0a57e0d5f> https://gist.github.com/cloudkitsch/eefec59233269ace0c9707d0a57e0d5f

However before diving into that I want to merely point out that the problem I’m having is not specific to my graphics configuration which works just fine except that HDMI audio doesn’t work. Not really sure why, maybe because I’m running Windows 10 N? Don’t know. Seems like I remember HDMI audio working on my Radeon 5770 / Opteron setup that I had VFIO setup on quite a while ago.

I don’t really want HDMI audio either. I have a Yamaha MG10-XU mixer connected to a USB 3 PCIe card that is also passed through to the guest. It is a USB audio DAC. The problem I’m having is very specifically with this mixer however the mixer works just fine on other computers. It doesn’t matter what buffer size I’m using @ 44.1khz (lowest sampling rate) I still have the same problem which is hard to explain because while I can reproduce it consistently I can’t understand the behavior. Asides from theories, I’m not really sure where I would start to test. I think that other people around me are more frustrated with it than I am but I would really like to first isolate the problem and provide my time (while available) to do what I can to eliminate the behavior or fix the problem.

I would like to suggest a few reasons why it might be happening, starting with the fact that the audio will start to skip when the load on the bare metal host increases or is saturated. This is not always the case, but I have not found a way to dedicate specific cores to the gust without having to do something really backwards with numactl to ensure nothing ever touches the cores that the guest is using. Maybe this is not worth trying anyway? It seems to suggest that the USB ASIO driver for the MG10-XU has issues with timing that are impacted by not enough CPU cycles.

It must be more than that though, because it seems to also cut out and skip relentlessly with mpeg (x264 specifically.) In a lot of games the sound tends to skip during cinematics and  some high bandwidth movies consistently cause this problem.

However, I can get audio playback up to 192khz FLAC working but the driver usually crashes after a while and usually requires the whole system to be pretty idle. The mixer driver doesn’t crash like that on other computers. Both the VGA card and USB card are situated in PCIe slots adjacent to the same CPU in the NUMA topology however I’m not making any changes to the NUMA configuration at runtime, I tried without finding that it made any difference.

I used to have another USB card but that one did the same thing. Everything else that I could possibly care about works fantastic, including the mouse responsiveness—I have a whole powered hub of USB devices including an xBox controller connected to the USB card all of which work great. Games work really well in 1080p at 120hz. I would try to pass the mixer through VIA USB redirection (same way I’m doing the corsair PSU) but it wouldn’t work and I don’t imagine it would be much better. I’ve also tried getting my Scarlett 2i2 with it but I have the same kind of problems and it typically doesn’t work for more than 20 seconds at a time.

Any thoughts or information you have to share would be appreciated. I’d be happy to try and debug this some more but I’m completely out of ideas of where to go. It’s an interesting problem to say the least. I have tried a a lot of different things except those that I can’t really try. Please get in touch if you have any ideas.

Thank you,

-Paige

_______________________________________________

vfio-users mailing list

 <mailto:vfio-users at redhat.com> vfio-users at redhat.com

 <https://www.redhat.com/mailman/listinfo/vfio-users> https://www.redhat.com/mailman/listinfo/vfio-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20170223/6321047b/attachment.htm>