<html> <head> <meta content="text/html; charset=utf-8" http-equiv="Content-Type"> </head> <body bgcolor="#FFFFFF" text="#000000"> This indeed may be kernel regression. I tested 3.10, 3.12, and 4.1 kernels. Valley benchmark gives 1500~ points on 3.10 while 3.12 and 4.1 get only 1300~ - a 200 point difference. With 3.10 kernel i also have better DPC, more yellow bars than with any other kernel. However it also makes things act weird. For example sometimes after boot it takes a while until UAC prompt starts to work. It freezes calling application for a looong time before it starts working. Can take 10 minutes or so. But after some time its ok. I wanted to try bf4 online but it would never load level (always silently crashes at some point) while single player worked. 3.12 and 4.1 kernels had no problem loading online session. Also it often freezes host on guest shutdown. All tested kernel versions are stock kernels without any patches (basically whatever comes from aur or archlinux repos). All this with following config: <blockquote type="cite"> <vcpu placement='static'>4</vcpu> <cputune> <vcpupin vcpu='0' cpuset='4'/> <vcpupin vcpu='1' cpuset='5'/> <vcpupin vcpu='2' cpuset='6'/> <vcpupin vcpu='3' cpuset='7'/> <emulatorpin cpuset='0-3'/> </cputune></blockquote> <blockquote type="cite"> <cpu mode='host-model'> <model fallback='allow'/> <topology sockets='1' cores='4' threads='1'/> </cpu></blockquote> <blockquote type="cite">cset set -c 0-3 system > /dev/null</blockquote> <blockquote type="cite">cset proc -m -f root -t system -k</blockquote> <blockquote type="cite">nohz_full=2,3,4,5,6,7 rcu_nocbs=2,3,4,5,6,7</blockquote> So we have a fast kernel that is glitchy and slower kernels that work reasonably well. What a luck.. <div class="moz-cite-prefix">On 2016.01.11 17:08, Quentin Deldycke wrote: </div> <blockquote cite="mid:CAHYLta7GjsoW4zGouNdZB38FVJtQ9bj8LRC-f0GAzGfh6ZskpA@mail.gmail.com" type="cite"> <div dir="ltr">Hello, <div> </div> <div>I use intel cpu (i7 4790k). But yes i have an R9 290 as gpu.</div> <div>I try to offload to core 0, so in fact i can keep threads 0 and 4 for linux</div> <div>The rest of your resume is right.</div> <div> </div> <div> </div> <div>I use the same program for dpc check. Is also available: latencymon.</div> <div>But i find dpclat more interesting.</div> <div><a moz-do-not-send="true" href="http://www.resplendence.com/latencymon" target="_blank">http://www.resplendence.com/latencymon</a> </div> <div> </div> <div>Script for moving all threads to 0,4: </div> <div><a moz-do-not-send="true" href="https://github.com/qdel/scripts/blob/master/vfio/shieldbuild">https://github.com/qdel/scripts/blob/master/vfio/shieldbuild </a></div> <div> </div> <div>XML file:</div> <div><a moz-do-not-send="true" href="https://github.com/qdel/scripts/blob/master/vfio/win10.xml">https://github.com/qdel/scripts/blob/master/vfio/win10.xml </a></div> <div> </div> <div>Kernel command line:</div> <div>intel_iommu=on iommu=pt vfio_iommu_type1.allow_unsafe_interrupts=1</div> <div>kvm.ignore_msrs=1 drm.rnodes=1 i915.modeset=1</div> <div>nohz_full=1,2,3,4,5,6,7 rcu_nocbs=1,2,3,4,5,6,7</div> <div>default_hugepagesz=1G hugepagesz=1G hugepages=12</div> <div> </div> <div> </div> <div>Notes about my setup:</div> <div>* I have 3 monitors. All are connected to intel</div> <div> 2 of them have a input in AMD. With xrandr</div> <div> i can disable these screens and they switch source</div> <div> (at least one, the other is buggy, most times i need to push the source button).</div> <div>* I pass also a NVMe drive (this thing is actually BRUTAL!!!)</div> <div> - I can boot the same drive on native!</div> <div>* I pass my second network card</div> <div>* I pass one of my sata controller (i have ntfs drives there)</div> <div>* I pass usb devices and not the whole controller</div> <div> - with the little udev script i plug new devices to the vm if this one is started</div> <div>* Sound is output by HDMI and back into line in of pc. I can use the line in control to</div> <div> modify the sound of the whole vm. Work perfectly actually.</div> <div>* Best thing for this is dual monitor + synergy :)</div> <div> </div> </div> <div class="gmail_extra"> <div> <div class="gmail_signature"> <div dir="ltr">-- <div>Deldycke Quentin </div> <div> <div> </div> </div> </div> </div> </div> <div class="gmail_quote">On 11 January 2016 at 15:06, Milos Kaurin <<a moz-do-not-send="true" href="mailto:milos.kaurin@gmail.com" target="_blank">milos.kaurin@gmail.com</a>> wrote: <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello, Yes, I have a corei7. I have to admit that seeing Quentin's e-mail was the first I found out about DPC latency. I'm taking a strictly empirical approach for now, but I'd like to dive deeper into this, at least to provide a reference point for you guys. Reason for this being is that even though I'm familiar with Linux, I'm don't have low-level familiarity as you guys have (other than conceptual). I'm more than willing to learn given the opportunity, though. Quentin: >From what I understand about your use: * You have an AMD CPU * In your kernel parameters, you are trying to offload your scheduling-clock interrupts to only thread(core?) 0. * Your script sets kernel memory management, future tasks and current tasks to be run at thread 0 * Valley bench seems to be most sensitive to DPC latency issues (as well as "Heroes of the storm") * Pinning only 3 cores to the VM gives you best results, but seeing that newer games take advantage of multiple cores, you'd like to have an option to use more cores for winVirt What I'd like from you: * Can you provide me with the optimal (3core -> VM) settings, including kernel parameters, your updated script and the XML of your virt in this mode of use. * Can you provide me with a method how to keep track of DPC latency? I found this: <a moz-do-not-send="true" href="http://www.thesycon.de/deu/latency_check.shtml" rel="noreferrer" target="_blank">http://www.thesycon.de/deu/latency_check.shtml</a> , but I'd like us to use the same method. Why I'm asking all of this: Just ran valley (HD extreme). These are the results: *Bare-metal: FPS: 48.7 Score: 2036 Min FPS: 23.4 Max FPS:90.6 * hugetables, nopin, 1x4x2, host-passthrough: FPS: 47.9 Score: 2005 Min FPS: 19.7 Max FPS: 91.5 The score is ~1.5 % worse in the virt. The min FPS difference (which looks significant) might be negligible because I'm running Firefox in the host with a bunch of tabs open (idle, though) I have also been playing "Rocket League" in the virt which is a very twitchy game, and I play it on an experienced level. I did not find any problems with playing the game like this. My current XML: <a moz-do-not-send="true" href="https://gist.github.com/Kaurin/0b6726e8a94084bd0b64" rel="noreferrer" target="_blank">https://gist.github.com/Kaurin/0b6726e8a94084bd0b64</a> PCI devices passed through: nvidia+HDMI audio, onboard sound, onboard XHCI USB controller Notes about my setup: * Both virt and host are hooked up to the same monitor (host-VGA / virt - DVI). * I also don't have any additional USB controllers, which means that when I turn on the virt, I lose my usb(mouse,keyboard) on the host * Same goes for sound: when I turn on the virt, I lose sound in the host * I just flip the monitor input and I'm good to go. * I have plans to set up new hardware so I can use both host/virt at the same time Let me know if my further input would be useful. Regards, Milos <div class="HOEnZb"> <div class="h5"> On Mon, Jan 11, 2016 at 9:19 AM, Quentin Deldycke <<a moz-do-not-send="true" href="mailto:quentindeldycke@gmail.com">quentindeldycke@gmail.com</a>> wrote: > In fact, some games react quite well to this latency. Fallout for example > doesn't show much difference between host - vm with brutal DPC and vm with > "good dpc". > > I tested 3 modes: > > - all 8 core to vm without pinning: brutal dpc, did not tried to play games > on it. Only ungine valley => 2600 points > - 6 cores pinned to the vm + emulator on core 0,1: correct latency. Most > games work flawlessly (bf4 / battlefront / diablo III) but some are > catastrophic: Heroes of the storm. valley => 2700 > - 3 cores pinned to vm: Perfect latency, all games work ok. But i am affraid > 3 cores are a bit 'not enough" for incoming games. valley => 3100 points > > I think that valley is a good benchmark. It is free and small. It seems to > be affected by this latency problem like most games. > > > > > -- > Deldycke Quentin > > > On 11 January 2016 at 09:59, rndbit <<a moz-do-not-send="true" href="mailto:rndbit@sysret.net"><a class="moz-txt-link-abbreviated" href="mailto:rndbit@sysret.net">rndbit@sysret.net</a></a>> wrote: >> >> Tried Milos' config too - DPC latency got worse. I use AMD cpu though so >> its hardly comparable. >> One thing to note is that both VM and bare metal (same OS) score around 5k >> points in 3dmark fire strike test (VM 300 points less). Sounds not too bad >> but in reality bf4 is pretty much unplayable in VM due to bad performance >> and sound glitches while playing it on bare metal is just fine. Again DPC >> latency on bare metal even under load is ok - occasional spike here and >> there but mostly its within norm. Any kind of load on VM makes DPC go nuts >> and performance is terrible. I even tried isolcpus=4,5,6,7 and binding vm to >> those free cores - its all the same. >> >> Interesting observation is that i used to play titanfall without a hitch >> in VM some time in the past, 3.10 kernel or so (no patches). When i get free >> moment ill try downgrading kernel, maybe problem is there. >> >> >> On 2016.01.11 10:39, Quentin Deldycke wrote: >> >> Also, i juste saw something: >> >> You use ultra (4k?) settings on a 770gtx. This is too heavy for it. You >> have less than 10fps. So in fact if you loose let's say 10% of performance, >> you will barely see it. >> >> What we search is a very high reponse time. Could you please compare your >> system with a less heavy benchmark. It is easier to see the difference at >> ~50-70 fps. >> >> In my case, this configuration work. But my fps fluctuate quite a lot. If >> you are a bit a serious gamer, this falls are not an option during game :) >> >> -- >> Deldycke Quentin >> >> >> On 11 January 2016 at 08:54, Quentin Deldycke <<a moz-do-not-send="true" href="mailto:quentindeldycke@gmail.com">quentindeldycke@gmail.com</a>> >> wrote: >>> >>> Using this mode, >>> >>> DPC Latency is hugely buggy using this mode. >>> >>> My fps are also moving on an apocaliptic way: from 80 to 45 fps without >>> moving on ungine valley. >>> >>> Do you have anything working on your linux? (i have plasma doing nothing >>> on another screen) >>> >>> Ungine heaven went back to 2600 points from 3100 >>> Cinebench r15: single core 124 >>> >>> >>> Could you please send your whole xml file, qemu version and kernel config >>> / boot? >>> >>> I will try to get 3dmark and verify host / virtual comparison >>> >>> -- >>> Deldycke Quentin >>> >>> >>> On 9 January 2016 at 20:24, Milos Kaurin <<a moz-do-not-send="true" href="mailto:milos.kaurin@gmail.com">milos.kaurin@gmail.com</a>> wrote: >>>> >>>> My details: >>>> Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz >>>> 32GB total ram >>>> hugetables@16x1GB for the guest (didn't have much to do with 3dmark >>>> results) >>>> >>>> I have had the best performance with: >>>> >>>> <vcpu placement='static'>8</vcpu> >>>> <cpu mode='custom' match='exact'> >>>> <model fallback='allow'>host-passthrough</model> >>>> <topology sockets='1' cores='4' threads='2'/> >>>> </cpu> >>>> >>>> No CPU pinning on either guest or host >>>> >>>> Benchmark example (Bare metal Win10 vs Fedora Guest Win10) >>>> <a moz-do-not-send="true" href="http://www.3dmark.com/compare/fs/7076732/fs/7076627#" rel="noreferrer" target="_blank">http://www.3dmark.com/compare/fs/7076732/fs/7076627#</a> >>>> >>>> >>>> Could you try my settings and report back? >>>> >>>> On Sat, Jan 9, 2016 at 3:14 PM, Quentin Deldycke >>>> <<a moz-do-not-send="true" href="mailto:quentindeldycke@gmail.com">quentindeldycke@gmail.com</a>> wrote: >>>> > I use virsh: >>>> > >>>> > ===SNIP=== >>>> > <vcpu placement='static'>3</vcpu> >>>> > <cputune> >>>> > <vcpupin vcpu='0' cpuset='1'/> >>>> > <vcpupin vcpu='1' cpuset='2'/> >>>> > <vcpupin vcpu='2' cpuset='3'/> >>>> > <emulatorpin cpuset='6-7'/> >>>> > </cputune> >>>> > ===SNAP=== >>>> > >>>> > I have a prepare script running: >>>> > >>>> > ===SNIP=== >>>> > sudo mkdir /cpuset >>>> > sudo mount -t cpuset none /cpuset/ >>>> > cd /cpuset >>>> > echo 0 | sudo tee -a cpuset.cpu_exclusive >>>> > echo 0 | sudo tee -a cpuset.mem_exclusive >>>> > >>>> > sudo mkdir sys >>>> > echo 'Building shield for core system... threads 0 and 4, and we place >>>> > all >>>> > runnning tasks there' >>>> > /bin/echo 0,4 | sudo tee -a sys/cpuset.cpus >>>> > /bin/echo 0 | sudo tee -a sys/cpuset.mems >>>> > /bin/echo 0 | sudo tee -a sys/cpuset.cpu_exclusive >>>> > /bin/echo 0 | sudo tee -a sys/cpuset.mem_exclusive >>>> > for T in `cat tasks`; do sudo bash -c "/bin/echo $T > >>>> > sys/tasks">/dev/null >>>> > 2>&1 ; done >>>> > cd - >>>> > ===SNAP=== >>>> > >>>> > Note that i use this command line for the kernel >>>> > nohz_full=1,2,3,4,5,6,7 rcu_nocbs=1,2,3,4,5,6,7 default_hugepagesz=1G >>>> > hugepagesz=1G hugepages=12 >>>> > >>>> > >>>> > -- >>>> > Deldycke Quentin >>>> > >>>> > >>>> > On 9 January 2016 at 15:40, rndbit <<a moz-do-not-send="true" href="mailto:rndbit@sysret.net">rndbit@sysret.net</a>> wrote: >>>> >> >>>> >> Mind posting actual commands how you achieved this? >>>> >> >>>> >> All im doing now is this: >>>> >> >>>> >> cset set -c 0-3 system >>>> >> cset proc -m -f root -t system -k >>>> >> >>>> >> <vcpu placement='static'>4</vcpu> >>>> >> <cputune> >>>> >> <vcpupin vcpu='0' cpuset='4'/> >>>> >> <vcpupin vcpu='1' cpuset='5'/> >>>> >> <vcpupin vcpu='2' cpuset='6'/> >>>> >> <vcpupin vcpu='3' cpuset='7'/> >>>> >> <emulatorpin cpuset='0-3'/> >>>> >> </cputune> >>>> >> >>>> >> Basically this puts most of threads to 0-3 cores including emulator >>>> >> threads. Some threads cant be moved though so they remain on 4-7 >>>> >> cores. VM >>>> >> is given 4-7 cores. It works better but there is still much to be >>>> >> desired. >>>> >> >>>> >> >>>> >> >>>> >> On 2016.01.09 15:59, Quentin Deldycke wrote: >>>> >> >>>> >> Hello, >>>> >> >>>> >> Using cpuset, i was using the vm with: >>>> >> >>>> >> Core 0: threads 0 & 4: linux + emulator pin >>>> >> Core 1,2,3: threads 1,2,3,5,6,7: windows >>>> >> >>>> >> I tested with: >>>> >> Core 0: threads 0 & 4: linux >>>> >> Core 1,2,3: threads 1,2,3: windows >>>> >> Core 1,2,3: threads 5,6,7: emulator >>>> >> >>>> >> The difference between both is huge (DPC latency is mush more >>>> >> stable): >>>> >> Performance on single core went up to 50% (cinebench ratio by core >>>> >> from >>>> >> 100 to 150 points) >>>> >> Performance on gpu went up to 20% (cinebench from 80fps to 100+) >>>> >> Performance on "heroes of the storm" went from 20~30 fps to stable 60 >>>> >> (and >>>> >> much time more than 100) >>>> >> >>>> >> (performance of Unigine Heaven went from 2700 points to 3100 points) >>>> >> >>>> >> The only sad thing is that i have the 3 idle threads which are barely >>>> >> used... Is there any way to put them back to windows? >>>> >> >>>> >> -- >>>> >> Deldycke Quentin >>>> >> >>>> >> >>>> >> On 29 December 2015 at 17:38, Michael Bauer <<a moz-do-not-send="true" href="mailto:michael@m-bauer.org">michael@m-bauer.org</a>> >>>> >> wrote: >>>> >>> >>>> >>> I noticed that attaching a DVD-Drive from the host leads to HUGE >>>> >>> delays. >>>> >>> I had attached my /dev/sr0 to the guest and even without a DVD in >>>> >>> the drive >>>> >>> this was causing huge lag about once per second. >>>> >>> >>>> >>> Best regards >>>> >>> Michael >>>> >>> >>>> >>> >>>> >>> Am 28.12.2015 um 19:30 schrieb rndbit: >>>> >>> >>>> >>> 4000μs-16000μs here, its terrible. >>>> >>> Tried whats said on >>>> >>> <a moz-do-not-send="true" href="https://lime-technology.com/forum/index.php?topic=43126.15" rel="noreferrer" target="_blank">https://lime-technology.com/forum/index.php?topic=43126.15</a> >>>> >>> Its a bit better with this: >>>> >>> >>>> >>> <vcpu placement='static'>4</vcpu> >>>> >>> <cputune> >>>> >>> <vcpupin vcpu='0' cpuset='4'/> >>>> >>> <vcpupin vcpu='1' cpuset='5'/> >>>> >>> <vcpupin vcpu='2' cpuset='6'/> >>>> >>> <vcpupin vcpu='3' cpuset='7'/> >>>> >>> <emulatorpin cpuset='0-3'/> >>>> >>> </cputune> >>>> >>> >>>> >>> I tried isolcpus but it did not yield visible benefits. ndis.sys is >>>> >>> big >>>> >>> offender here but i dont really understand why. Removing network >>>> >>> interface >>>> >>> from VM makes usbport.sys take over as biggest offender. All this >>>> >>> happens >>>> >>> with performance governor of all cpu cores: >>>> >>> >>>> >>> echo performance | tee >>>> >>> /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor >/dev/null >>>> >>> >>>> >>> Cores remain clocked at 4k mhz. I dont know what else i could try. >>>> >>> Does >>>> >>> anyone have any ideas..? >>>> >>> >>>> >>> On 2015.10.29 08:03, Eddie Yen wrote: >>>> >>> >>>> >>> I tested again with VM reboot, I found that this time is about >>>> >>> 1000~1500μs. >>>> >>> Also I found that it easily get high while hard drive is loading, >>>> >>> but >>>> >>> only few times. >>>> >>> >>>> >>> Which specs you're using? Maybe it depends on CPU or patches. >>>> >>> >>>> >>> 2015-10-29 13:44 GMT+08:00 Blank Field <<a moz-do-not-send="true" href="mailto:ihatethisfield@gmail.com">ihatethisfield@gmail.com</a>>: >>>> >>>> >>>> >>>> If i understand it right, this software has a fixed latency error >>>> >>>> of 1 >>>> >>>> ms(1000us) in windows 8-10 due to different kernel timer >>>> >>>> implementation. So >>>> >>>> i guess your latency is very good. >>>> >>>> >>>> >>>> On Oct 29, 2015 8:40 AM, "Eddie Yen" <<a moz-do-not-send="true" href="mailto:missile0407@gmail.com">missile0407@gmail.com</a>> wrote: >>>> >>>>> >>>> >>>>> Thanks for information! And sorry I don'r read carefully at >>>> >>>>> beginning >>>> >>>>> message. >>>> >>>>> >>>> >>>>> For my result, I got about 1000μs below and only few times got >>>> >>>>> 1000μs >>>> >>>>> above when idling. >>>> >>>>> >>>> >>>>> I'm using 4820K and used 4 threads to VM, also I set these 4 >>>> >>>>> threads >>>> >>>>> as 4 cores in VM settings. >>>> >>>>> The OS is Windows 10. >>>> >>>>> >>>> >>>>> 2015-10-29 13:21 GMT+08:00 Blank Field <<a moz-do-not-send="true" href="mailto:ihatethisfield@gmail.com">ihatethisfield@gmail.com</a>>: >>>> >>>>>> >>>> >>>>>> I think they're using this: >>>> >>>>>> <a moz-do-not-send="true" href="http://www.thesycon.de/deu/latency_check.shtml" rel="noreferrer" target="_blank"><a class="moz-txt-link-abbreviated" href="http://www.thesycon.de/deu/latency_check.shtml">www.thesycon.de/deu/latency_check.shtml</a></a> >>>> >>>>>> >>>> >>>>>> On Oct 29, 2015 6:11 AM, "Eddie Yen" <<a moz-do-not-send="true" href="mailto:missile0407@gmail.com">missile0407@gmail.com</a>> >>>> >>>>>> wrote: >>>> >>>>>>> >>>> >>>>>>> Sorry, but how to check DPC Latency? >>>> >>>>>>> >>>> >>>>>>> 2015-10-29 10:08 GMT+08:00 Nick Sukharev >>>> >>>>>>> <<a moz-do-not-send="true" href="mailto:nicksukharev@gmail.com"><a class="moz-txt-link-abbreviated" href="mailto:nicksukharev@gmail.com">nicksukharev@gmail.com</a></a>>: >>>> >>>>>>>> >>>> >>>>>>>> I just checked on W7 and I get 3000μs-4000μs one one of the >>>> >>>>>>>> guests >>>> >>>>>>>> when 3 guests are running. >>>> >>>>>>>> >>>> >>>>>>>> On Wed, Oct 28, 2015 at 4:52 AM, Sergey Vlasov >>>> >>>>>>>> <<a moz-do-not-send="true" href="mailto:sergey@vlasov.me"><a class="moz-txt-link-abbreviated" href="mailto:sergey@vlasov.me">sergey@vlasov.me</a></a>> >>>> >>>>>>>> wrote: >>>> >>>>>>>>> >>>> >>>>>>>>> On 27 October 2015 at 18:38, LordZiru <<a moz-do-not-send="true" href="mailto:lordziru@gmail.com"><a class="moz-txt-link-abbreviated" href="mailto:lordziru@gmail.com">lordziru@gmail.com</a></a>> >>>> >>>>>>>>> wrote: >>>> >>>>>>>>>> >>>> >>>>>>>>>> I have brutal DPC Latency on qemu, no matter if using >>>> >>>>>>>>>> pci-assign >>>> >>>>>>>>>> or vfio-pci or without any passthrought, >>>> >>>>>>>>>> >>>> >>>>>>>>>> my DPC Latency is like: >>>> >>>>>>>>>> 10000,500,8000,6000,800,300,12000,9000,700,2000,9000 >>>> >>>>>>>>>> and on native windows 7 is like: >>>> >>>>>>>>>> 20,30,20,50,20,30,20,20,30 >>>> >>>>>>>>> >>>> >>>>>>>>> >>>> >>>>>>>>> In Windows 10 guest I constantly have red bars around 3000μs >>>> >>>>>>>>> (microseconds), spiking sometimes up to 10000μs. >>>> >>>>>>>>> >>>> >>>>>>>>>> >>>> >>>>>>>>>> I don't know how to fix it. >>>> >>>>>>>>>> this matter for me because i are using USB Sound Card for my >>>> >>>>>>>>>> VMs, >>>> >>>>>>>>>> and i get sound drop-outs every 0-4 secounds >>>> >>>>>>>>>> >>>> >>>>>>>>> >>>> >>>>>>>>> That bugs me a lot too. I also use an external USB card and my >>>> >>>>>>>>> DAW >>>> >>>>>>>>> periodically drops out :( >>>> >>>>>>>>> >>>> >>>>>>>>> I haven't tried CPU pinning yet though. And perhaps I should >>>> >>>>>>>>> try >>>> >>>>>>>>> Windows 7. >>>> >>>>>>>>> >>>> >>>>>>>>> >>>> >>>>>>>>> _______________________________________________ >>>> >>>>>>>>> vfio-users mailing list >>>> >>>>>>>>> <a moz-do-not-send="true" href="mailto:vfio-users@redhat.com"><a class="moz-txt-link-abbreviated" href="mailto:vfio-users@redhat.com">vfio-users@redhat.com</a></a> >>>> >>>>>>>>> <a moz-do-not-send="true" href="https://www.redhat.com/mailman/listinfo/vfio-users" rel="noreferrer" target="_blank"><a class="moz-txt-link-freetext" href="https://www.redhat.com/mailman/listinfo/vfio-users">https://www.redhat.com/mailman/listinfo/vfio-users</a></a> >>>> >>>>>>>>> >>>> >>>>>>>> >>>> >>>>>>>> >>>> >>>>>>>> _______________________________________________ >>>> >>>>>>>> vfio-users mailing list >>>> >>>>>>>> <a moz-do-not-send="true" href="mailto:vfio-users@redhat.com"><a class="moz-txt-link-abbreviated" href="mailto:vfio-users@redhat.com">vfio-users@redhat.com</a></a> >>>> >>>>>>>> <a moz-do-not-send="true" href="https://www.redhat.com/mailman/listinfo/vfio-users" rel="noreferrer" target="_blank"><a class="moz-txt-link-freetext" href="https://www.redhat.com/mailman/listinfo/vfio-users">https://www.redhat.com/mailman/listinfo/vfio-users</a></a> >>>> >>>>>>>> >>>> >>>>>>> >>>> >>>>>>> >>>> >>>>>>> _______________________________________________ >>>> >>>>>>> vfio-users mailing list >>>> >>>>>>> <a moz-do-not-send="true" href="mailto:vfio-users@redhat.com"><a class="moz-txt-link-abbreviated" href="mailto:vfio-users@redhat.com">vfio-users@redhat.com</a></a> >>>> >>>>>>> <a moz-do-not-send="true" href="https://www.redhat.com/mailman/listinfo/vfio-users" rel="noreferrer" target="_blank"><a class="moz-txt-link-freetext" href="https://www.redhat.com/mailman/listinfo/vfio-users">https://www.redhat.com/mailman/listinfo/vfio-users</a></a> >>>> >>>>>>> >>>> >>>>> >>>> >>> >>>> >>> >>>> >>> >>>> >>> _______________________________________________ >>>> >>> vfio-users mailing list >>>> >>> <a moz-do-not-send="true" href="mailto:vfio-users@redhat.com">vfio-users@redhat.com</a> >>>> >>> <a moz-do-not-send="true" href="https://www.redhat.com/mailman/listinfo/vfio-users" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/vfio-users</a> >>>> >>> >>>> >>> >>>> >>> >>>> >>> >>>> >>> _______________________________________________ >>>> >>> vfio-users mailing list >>>> >>> <a moz-do-not-send="true" href="mailto:vfio-users@redhat.com">vfio-users@redhat.com</a> >>>> >>> <a moz-do-not-send="true" href="https://www.redhat.com/mailman/listinfo/vfio-users" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/vfio-users</a> >>>> >>> >>>> >>> >>>> >>> >>>> >>> _______________________________________________ >>>> >>> vfio-users mailing list >>>> >>> <a moz-do-not-send="true" href="mailto:vfio-users@redhat.com">vfio-users@redhat.com</a> >>>> >>> <a moz-do-not-send="true" href="https://www.redhat.com/mailman/listinfo/vfio-users" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/vfio-users</a> >>>> >>> >>>> >> >>>> >> >>>> >> >>>> >> _______________________________________________ >>>> >> vfio-users mailing list >>>> >> <a moz-do-not-send="true" href="mailto:vfio-users@redhat.com">vfio-users@redhat.com</a> >>>> >> <a moz-do-not-send="true" href="https://www.redhat.com/mailman/listinfo/vfio-users" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/vfio-users</a> >>>> >> >>>> >> >>>> >> >>>> >> _______________________________________________ >>>> >> vfio-users mailing list >>>> >> <a moz-do-not-send="true" href="mailto:vfio-users@redhat.com">vfio-users@redhat.com</a> >>>> >> <a moz-do-not-send="true" href="https://www.redhat.com/mailman/listinfo/vfio-users" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/vfio-users</a> >>>> >> >>>> > >>>> > >>>> > _______________________________________________ >>>> > vfio-users mailing list >>>> > <a moz-do-not-send="true" href="mailto:vfio-users@redhat.com">vfio-users@redhat.com</a> >>>> > <a moz-do-not-send="true" href="https://www.redhat.com/mailman/listinfo/vfio-users" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/vfio-users</a> >>>> > >>> >>> >> >> >> >> _______________________________________________ >> vfio-users mailing list >> <a moz-do-not-send="true" href="mailto:vfio-users@redhat.com">vfio-users@redhat.com</a> >> <a moz-do-not-send="true" href="https://www.redhat.com/mailman/listinfo/vfio-users" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/vfio-users</a> >> >> >> >> _______________________________________________ >> vfio-users mailing list >> <a moz-do-not-send="true" href="mailto:vfio-users@redhat.com">vfio-users@redhat.com</a> >> <a moz-do-not-send="true" href="https://www.redhat.com/mailman/listinfo/vfio-users" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/vfio-users</a> >> > > > _______________________________________________ > vfio-users mailing list > <a moz-do-not-send="true" href="mailto:vfio-users@redhat.com">vfio-users@redhat.com</a> > <a moz-do-not-send="true" href="https://www.redhat.com/mailman/listinfo/vfio-users" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/vfio-users</a> > </div> </div> </blockquote> </div> </div> </blockquote> </body> </html>