[vfio-users] Minimal host with desktop guest and gaming guest. Is this possible?

[Alex Williamson]

On Sun, 16 Jul 2017 12:59:16 +0200
almer <almertje at gmail.com> wrote:

> > Yes, it's possible, not even hard.  
> 
> Ok, so this isn't a hazy unattainable dream, perfect!
> 
> 
> > unbind cards or unload the module and RX 480  
> 
> The host would get the IGP from the CPU. Both PCIE cards would be
> vfio-stubed, so as far as I understand it these cards will be handled by
> the vfio module in the host kernel and should be freely assignable to
> guests. Therefore don't use any power if no guest is using it.

Um, not quite.  Unless you have hotplug capable slots with independent
power control, the best we can do is put devices into a "low power", D3
power state.  The PCI spec requires devices to support this state, but
does not require devices to meet any particular standards for power
consumption in this state.  I would characterize D3 as slightly better
than doing nothing, no more.  In fact, the power consumption might be
lower with the VM running and the guest driver managing the device
specific power controls.

> I'm not deadset on Nvidia I rather would use something else because of the
> uncertainty that comes with and Nvidia driver. My guess is that their
> proprietary Nvidia linux driver would have VM detection too, but didn't
> find confirmation on this. I would like to use a AMD graphics card just to
> be safe and out of that armsrace, AMDs driver doesn't block like Nvidia
> does.
> But I got discouraged by AMD graphics card PCIE reset issue reports and I
> couldn't make heads and tails about it. If any RX 480 works, great. I will
> have to look more into this.
> 
> 
> > My Linux desktop get the IGD [..] Why not make more use of IGD assignment
> > in your configuration.  
> 
> Does a host system even boot with all graphics devices stubed? So am I
> right to assuming that x86 "graphic needs" are satisfied for POST but after
> that doesn't care what happens to the graphics devices afterwards? I don't
> mind a headless host at all, it will change somewhat how I setup things for
> crypto/storage but that I can solve.

Linux doesn't require a graphics head, I use a serial console for my
system.  I personally don't like entirely headless configurations (ie.
no graphical or serial console), but to each their own.

> For the IGP to be stubed and passed through I need a ACS capable CPU? As it
> seems I wrongfully assumed that Vt-d and Vt-x support entailed this.

IGD is not dependent on ACS, IGD is a single function device connected
directly to the PCIe root complex.  It will always be in an IOMMU group
of its own.  Unfortunately the Intel systems that include proper ACS do
not include IGD, and conversely, if you have IGD you do not have ACS.
Plan your usage accordingly.
 
> > Concerns with your hardware choice otherwise: a) No ACS, you won't be  
> able
> > to assign cards in CPU root port slots to separate VMs
> > (see vfio.blogspot.com), b) insufficient cores, how much are you
> > willing to have your storage server interfere with your gaming  
> performance?
> 
> a) Ok, I found your article http://vfio.blogspot.de/2015/
> 10/intel-processors-with-acs-support.html these are the only choices?

At the time of writing, I think it was fairly comprehensive, since
Intels product SKU marketing scramble with "Scalable Processors" I'm
not sure we've figured out the new decoder ring yet.

> Doesn't AMD support this too? AMD CPUs are usually weaker in single core
> performance, so I ignored AMD CPUs for years. But seeing the prices I
> wonder if they have an alternative for ~450 Euro price range. Some of the
> Intel CPUs you listed are way out of my budget. The strongest and somewhat
> affordable listed is a i7-6850k. AMD ryzen 1800x costs ~470 Euro, but
> doesn't have an IGP and I didn't look into AMD CPUs at all so I don't know
> if they support ACS it has AMD-V tho.

Previous generation, 990-based AM3/+ sockets supported ACS.  Ryzen
systems seem to have implemented it in hardware, but nobody told the
BIOS team that it was important to enable and support for ACS is slowly
being added.  Specifications are hard to come by for AMD chips though,
so we only have user reports to tell us how various systems fair.
Increased core count in Ryzen is certainly desirable for virtualization
scenarios, but the number of PCIe lanes from the processor can lead to
limited I/O options or additional switch components that may ruin the
ACS provided by the root ports.  Note that there's also an assigned GPU
performance issue with AMD's NPT as well that isn't understood.
 
> b) Well. I'm used to Intel quad core CPUs. Seeing that no matter what I buy
> I will endup with 6 cores at least, I guess I can live with it. Desktop
> guest and host heavyload-PIDs will get taskset to one core and the gaming
> guest gets the rest pinned and taskset. So my guess is that any
> crypt/storage/desktop spike shouldn't be noticed in the gaming guest.

6-cores?  At best you can get 4-cores + threads on Intel processors
with IGD.  Folks here tend to complain about every little micro jitter
they see in VM configurations and those can be difficult to eliminate
entirely without cores that can be dedicated to the gaming VM.

> > The only major issue is that running things involving sound and video  
> becomes
> > not possible on such desktop. But those can be viewed/listened locally, on
> > whatever actual machine you happen to be at. (Or also there are ways to  
> pass
> > through sound from a remote desktop, or use video-efficient protocols  
> such as
> > SPICE, but for my use I did not look into those as of yet).  
> 
> This would be an issue. I didn't look into audio and microphone that much.
> I would like to hear the audio from desktop guest and gaming guest and have
> microphone available at least in the gaming guest for voice chat. Right now
> this doesn't matter that much. Host and guests will be all GNU/Linux
> distributions so without having looked into it further I guess there is a
> server solution if any audio/mic passthrough scenario wouldn't work. I
> guess your SPICE suggestion is such a solution.
> Monitor wise I initially thought about HDMI switching host and desktop
> guest, but as it seems that I can vfio-stub all graphics devices with a
> proper CPU, I won't need the passive PCIE card in the case of i7-6850k
> anymore and can use the IGP for the desktop guest instead. The desktop
> guest will be dual monitor setup and the gaming guest will be switched on
> one of the monitors, like I have it right now with separate computers.
> Keyboard + mouse, I already use a two port usb switch. I guess I can still
> use that to switch between desktop and gaming guests, after one of the two
> usb-switch cables is passed through to the desktop guest after boot.
> 
> I'm exited and a bit confused.
> 
> How do you see if a CPU does support proper isolation or not? It isn't the
> feature list like AMD-V or Vt-d/x only. Is it a combination of mainboard
> chipset and CPU? I ask because I see the TDP wattage difference between
> ryzen 7 1800x and the i7-6850k. So being relatively equal in price and
> performance I tend towards the ryzen CPU. In the last couple hours I fail
> to find if it would support proper PCIE isolation or not. In the search
> stumbled upon ACS kernel patch, if possible I don't want to apply the ACS
> patch to the host kernel.

The ACS override patch is unsupported and not destined for upstream, so
I would certainly not use it.  ACS is typically included in the
register definition in the vol2 Intel datasheet.  For AMD, good luck,
we rely on user reports due to lack of published specs.  We know
that AGESA 1.0.0.6 is required for Ryzen root port ACS and I see
on another list that 1.0.0.7 is claimed will enable ACS on downstream
switch ports for an internal switch in X370.  If we can get to the
bottom of the NPT performance issue, Ryzen may yet shape up to be a
good device assignment platform, but it's taking its time to get
there.  Thanks,

Alex