[vfio-users] Minimal host with desktop guest and gaming guest. Is this possible?

[hristo at hiliev.eu]

Hi,

July 17, 2017 11:52 PM, "almer" <almertje at gmail.com> wrote:

> Thank you for clarifying!
> 
>>>> My Linux desktop get the IGD [..] Why not make more use of IGD
>>>> assignment in your configuration.
>>> 
>>> Does a host system even boot with all graphics devices stubed? So am I
>>> right to assuming that x86 "graphic needs" are satisfied for POST but
>>> after that doesn't care what happens to the graphics devices
>>> afterwards? I don't mind a headless host at all, it will change
>>> somewhat how I setup things for crypto/storage but that I can solve.
>> 
>> Linux doesn't require a graphics head, I use a serial console for my
>> system. I personally don't like entirely headless configurations (ie.
>> no graphical or serial console), but to each their own.
> 
> Well I like graphical output too, but for getting this to work I wouldn't
> mind missing it. For debugging I can always change in GRUB and start the
> host with a display or ssh into the host from a laptop. In the end I hope
> that a minimal host should run smoothly enough.
> 
>>>> Concerns with your hardware choice otherwise: a) No ACS, you won't
>>>> be able to assign cards in CPU root port slots to separate VMs (see
>>>> vfio.blogspot.com), b) insufficient cores, how much are you willing
>>>> to have your storage server interfere with your gaming performance?
>>> 
>>> a) Ok, I found your article
>>> http://vfio.blogspot.de/2015/10/intel-processors-with-acs-support.html
>>> these are the only choices?
>> 
>> At the time of writing, I think it was fairly comprehensive, since
>> Intels product SKU marketing scramble with "Scalable Processors" I'm not
>> sure we've figured out the new decoder ring yet.
> 
> Thank you for the blog articles. I'm still reading and it does clarify a
> lot, where I was confused before!
> 
>> 6-cores? At best you can get 4-cores + threads on Intel processors with
>> IGD. Folks here tend to complain about every little micro jitter they
>> see in VM configurations and those can be difficult to eliminate
>> entirely without cores that can be dedicated to the gaming VM.
> 
> You are right, I misread in my sea of browser tabs. Also misread the
> prices for a i7-6800k too, ~375 Euro that I can afford :) For the
> mainboard the Gigabyte GA-X99-UD4 has enough sata and usb3 for my
> storage/backup needs. From the manual I gather that it supports VT-d
> but I struggle to find a system block diagram to verify PCIE root
> ports to the pch and socket.
> http://www.gigabyte.us/Motherboard/GA-X99-UD4-rev-10#support-manual
> 

I have a GA-X99-UD4 motherboard with an i7-5820k and two NVidia GPUs,
a GT 730 for the host and a GTX 970 for the VMs. It works like charm.
Besides the GTX 970, I'm also passing the integrated Intel HDA sound
controller through. The host is running the latest stock Arch Linux
kernel. No ACS patches are needed as all (most?) LGA2011-3 CPUs provide
proper isolation.

There is a block diagram on page 8 of the manual, which shows that all
x8 and x16 slots are driven directly by the CPU and only the x1 slots
are driven by the PCH. If you have a CPU with only 28 PCIe lanes, e.g.
i7-5820k or i7-6800k, only one x16 slot functions as true x16. The
other x16 slot becomes x8 and the third x16 (physically x8) becomes x4.
Mind that the true x16 slot is the one closest to the CPU (PCIE_1 on
p. 7) and in my case an oversized CPU fan is almost touching the GPU.

I could provide you with the output of lspci -tv and the IOMMU
grouping in /sys/kernel/iommu_groups if necessary.

Earlier I was getting an "unstable TSC" problem, but flashing the F22
UEFI version seems to have fixed it.

By the way, I was trying to run my Linux desktop in yet another VM, but
I couldn't get the GT 730 to properly pass through and have the host
headless. Not that I've tried hard enough though.

Cheers,
Hristo

>> The ACS override patch is unsupported and not destined for upstream, so
>> I would certainly not use it. ACS is typically included in the register
>> definition in the vol2 Intel datasheet. For AMD, good luck, we rely on
>> user reports due to lack of published specs. We know that AGESA 1.0.0.6
>> is required for Ryzen root port ACS and I see on another list that
>> 1.0.0.7 is claimed will enable ACS on downstream switch ports for an
>> internal switch in X370. If we can get to the bottom of the NPT
>> performance issue, Ryzen may yet shape up to be a good device assignment
>> platform, but it's taking its time to get there. Thanks,
> 
> Ohh, so, Ryzen does have ACS but "starter" problems. I guess as a VM
> beginner/novice I'm better off sticking to older hardware like i7-6800k and
> GA-x99-UD4 and try to get my setup running.
> 
> Thanks and regards,
> Almer
> 
> On 17 July 2017 at 18:47, Alex Williamson <alex.williamson at redhat.com> wrote:
> 
>> On Sun, 16 Jul 2017 12:59:16 +0200
>> almer <almertje at gmail.com> wrote:
>> 
>>>> Yes, it's possible, not even hard.
>>> 
>>> Ok, so this isn't a hazy unattainable dream, perfect!
>>> 
>>> 
>>>> unbind cards or unload the module and RX 480
>>> 
>>> The host would get the IGP from the CPU. Both PCIE cards would be
>>> vfio-stubed, so as far as I understand it these cards will be handled by
>>> the vfio module in the host kernel and should be freely assignable to
>>> guests. Therefore don't use any power if no guest is using it.
>> 
>> Um, not quite. Unless you have hotplug capable slots with independent
>> power control, the best we can do is put devices into a "low power", D3
>> power state. The PCI spec requires devices to support this state, but
>> does not require devices to meet any particular standards for power
>> consumption in this state. I would characterize D3 as slightly better
>> than doing nothing, no more. In fact, the power consumption might be
>> lower with the VM running and the guest driver managing the device
>> specific power controls.
>> 
>>> I'm not deadset on Nvidia I rather would use something else because of the
>>> uncertainty that comes with and Nvidia driver. My guess is that their
>>> proprietary Nvidia linux driver would have VM detection too, but didn't
>>> find confirmation on this. I would like to use a AMD graphics card just to
>>> be safe and out of that armsrace, AMDs driver doesn't block like Nvidia
>>> does.
>>> But I got discouraged by AMD graphics card PCIE reset issue reports and I
>>> couldn't make heads and tails about it. If any RX 480 works, great. I will
>>> have to look more into this.
>>> 
>>> 
>>>> My Linux desktop get the IGD [..] Why not make more use of IGD assignment
>>>> in your configuration.
>>> 
>>> Does a host system even boot with all graphics devices stubed? So am I
>>> right to assuming that x86 "graphic needs" are satisfied for POST but after
>>> that doesn't care what happens to the graphics devices afterwards? I don't
>>> mind a headless host at all, it will change somewhat how I setup things for
>>> crypto/storage but that I can solve.
>> 
>> Linux doesn't require a graphics head, I use a serial console for my
>> system. I personally don't like entirely headless configurations (ie.
>> no graphical or serial console), but to each their own.
>> 
>>> For the IGP to be stubed and passed through I need a ACS capable CPU? As it
>>> seems I wrongfully assumed that Vt-d and Vt-x support entailed this.
>> 
>> IGD is not dependent on ACS, IGD is a single function device connected
>> directly to the PCIe root complex. It will always be in an IOMMU group
>> of its own. Unfortunately the Intel systems that include proper ACS do
>> not include IGD, and conversely, if you have IGD you do not have ACS.
>> Plan your usage accordingly.
>> 
>>>> Concerns with your hardware choice otherwise: a) No ACS, you won't be
>>> able
>>>> to assign cards in CPU root port slots to separate VMs
>>>> (see vfio.blogspot.com), b) insufficient cores, how much are you
>>>> willing to have your storage server interfere with your gaming
>>> performance?
>>> 
>>> a) Ok, I found your article http://vfio.blogspot.de/2015/
>>> 10/intel-processors-with-acs-support.html these are the only choices?
>> 
>> At the time of writing, I think it was fairly comprehensive, since
>> Intels product SKU marketing scramble with "Scalable Processors" I'm
>> not sure we've figured out the new decoder ring yet.
>> 
>>> Doesn't AMD support this too? AMD CPUs are usually weaker in single core
>>> performance, so I ignored AMD CPUs for years. But seeing the prices I
>>> wonder if they have an alternative for ~450 Euro price range. Some of the
>>> Intel CPUs you listed are way out of my budget. The strongest and somewhat
>>> affordable listed is a i7-6850k. AMD ryzen 1800x costs ~470 Euro, but
>>> doesn't have an IGP and I didn't look into AMD CPUs at all so I don't know
>>> if they support ACS it has AMD-V tho.
>> 
>> Previous generation, 990-based AM3/+ sockets supported ACS. Ryzen
>> systems seem to have implemented it in hardware, but nobody told the
>> BIOS team that it was important to enable and support for ACS is slowly
>> being added. Specifications are hard to come by for AMD chips though,
>> so we only have user reports to tell us how various systems fair.
>> Increased core count in Ryzen is certainly desirable for virtualization
>> scenarios, but the number of PCIe lanes from the processor can lead to
>> limited I/O options or additional switch components that may ruin the
>> ACS provided by the root ports. Note that there's also an assigned GPU
>> performance issue with AMD's NPT as well that isn't understood.
>> 
>>> b) Well. I'm used to Intel quad core CPUs. Seeing that no matter what I buy
>>> I will endup with 6 cores at least, I guess I can live with it. Desktop
>>> guest and host heavyload-PIDs will get taskset to one core and the gaming
>>> guest gets the rest pinned and taskset. So my guess is that any
>>> crypt/storage/desktop spike shouldn't be noticed in the gaming guest.
>> 
>> 6-cores? At best you can get 4-cores + threads on Intel processors
>> with IGD. Folks here tend to complain about every little micro jitter
>> they see in VM configurations and those can be difficult to eliminate
>> entirely without cores that can be dedicated to the gaming VM.
>> 
>>>> The only major issue is that running things involving sound and video
>>> becomes
>>>> not possible on such desktop. But those can be viewed/listened locally, on
>>>> whatever actual machine you happen to be at. (Or also there are ways to
>>> pass
>>>> through sound from a remote desktop, or use video-efficient protocols
>>> such as
>>>> SPICE, but for my use I did not look into those as of yet).
>>> 
>>> This would be an issue. I didn't look into audio and microphone that much.
>>> I would like to hear the audio from desktop guest and gaming guest and have
>>> microphone available at least in the gaming guest for voice chat. Right now
>>> this doesn't matter that much. Host and guests will be all GNU/Linux
>>> distributions so without having looked into it further I guess there is a
>>> server solution if any audio/mic passthrough scenario wouldn't work. I
>>> guess your SPICE suggestion is such a solution.
>>> Monitor wise I initially thought about HDMI switching host and desktop
>>> guest, but as it seems that I can vfio-stub all graphics devices with a
>>> proper CPU, I won't need the passive PCIE card in the case of i7-6850k
>>> anymore and can use the IGP for the desktop guest instead. The desktop
>>> guest will be dual monitor setup and the gaming guest will be switched on
>>> one of the monitors, like I have it right now with separate computers.
>>> Keyboard + mouse, I already use a two port usb switch. I guess I can still
>>> use that to switch between desktop and gaming guests, after one of the two
>>> usb-switch cables is passed through to the desktop guest after boot.
>>> 
>>> I'm exited and a bit confused.
>>> 
>>> How do you see if a CPU does support proper isolation or not? It isn't the
>>> feature list like AMD-V or Vt-d/x only. Is it a combination of mainboard
>>> chipset and CPU? I ask because I see the TDP wattage difference between
>>> ryzen 7 1800x and the i7-6850k. So being relatively equal in price and
>>> performance I tend towards the ryzen CPU. In the last couple hours I fail
>>> to find if it would support proper PCIE isolation or not. In the search
>>> stumbled upon ACS kernel patch, if possible I don't want to apply the ACS
>>> patch to the host kernel.
>> 
>> The ACS override patch is unsupported and not destined for upstream, so
>> I would certainly not use it. ACS is typically included in the
>> register definition in the vol2 Intel datasheet. For AMD, good luck,
>> we rely on user reports due to lack of published specs. We know
>> that AGESA 1.0.0.6 is required for Ryzen root port ACS and I see
>> on another list that 1.0.0.7 is claimed will enable ACS on downstream
>> switch ports for an internal switch in X370. If we can get to the
>> bottom of the NPT performance issue, Ryzen may yet shape up to be a
>> good device assignment platform, but it's taking its time to get
>> there. Thanks,
>> 
>> Alex