[vfio-users] Minimal host with desktop guest and gaming guest. Is this possible?

[almer]

Thank you for clarifying!

> > > My Linux desktop get the IGD [..] Why not make more use of IGD
> > > assignment in your configuration.
> >
> > Does a host system even boot with all graphics devices stubed? So am I
> > right to assuming that x86 "graphic needs" are satisfied for POST but
> > after that doesn't care what happens to the graphics devices
> > afterwards?  I don't mind a headless host at all, it will change
> > somewhat how I setup things for crypto/storage but that I can solve.
>
> Linux doesn't require a graphics head, I use a serial console for my
> system.  I personally don't like entirely headless configurations (ie.
> no graphical or serial console), but to each their own.

Well I like graphical output too, but for getting this to work I wouldn't
mind missing it. For debugging I can always change in GRUB and start the
host with a display or ssh into the host from a laptop. In the end I hope
that a minimal host should run smoothly enough.

> > > Concerns with your hardware choice otherwise: a) No ACS, you won't
> > > be able to assign cards in CPU root port slots to separate VMs (see
> > > vfio.blogspot.com), b) insufficient cores, how much are you willing
> > > to have your storage server interfere with your gaming performance?
> >
> > a) Ok, I found your article
> > http://vfio.blogspot.de/2015/10/intel-processors-with-acs-support.html
> > these are the only choices?
>
> At the time of writing, I think it was fairly comprehensive, since
> Intels product SKU marketing scramble with "Scalable Processors" I'm not
> sure we've figured out the new decoder ring yet.

Thank you for the blog articles. I'm still reading and it does clarify a
lot, where I was confused before!

> 6-cores?  At best you can get 4-cores + threads on Intel processors with
> IGD.  Folks here tend to complain about every little micro jitter they
> see in VM configurations and those can be difficult to eliminate
> entirely without cores that can be dedicated to the gaming VM.

You are right, I misread in my sea of browser tabs. Also misread the
prices for a i7-6800k too, ~375 Euro that I can afford :) For the
mainboard the Gigabyte GA-X99-UD4 has enough sata and usb3 for my
storage/backup needs. From the manual I gather that it supports VT-d
but I struggle to find a system block diagram to verify PCIE root
ports to the pch and socket.
http://www.gigabyte.us/Motherboard/GA-X99-UD4-rev-10#support-manual

> The ACS override patch is unsupported and not destined for upstream, so
> I would certainly not use it.  ACS is typically included in the register
> definition in the vol2 Intel datasheet.  For AMD, good luck, we rely on
> user reports due to lack of published specs.  We know that AGESA 1.0.0.6
> is required for Ryzen root port ACS and I see on another list that
> 1.0.0.7 is claimed will enable ACS on downstream switch ports for an
> internal switch in X370.  If we can get to the bottom of the NPT
> performance issue, Ryzen may yet shape up to be a good device assignment
> platform, but it's taking its time to get there.  Thanks,

Ohh, so, Ryzen does have ACS but "starter" problems. I guess as a VM
beginner/novice I'm better off sticking to older hardware like i7-6800k and
GA-x99-UD4 and try to get my setup running.

Thanks and regards,
Almer

On 17 July 2017 at 18:47, Alex Williamson <alex.williamson at redhat.com>
wrote:

> On Sun, 16 Jul 2017 12:59:16 +0200
> almer <almertje at gmail.com> wrote:
>
> > > Yes, it's possible, not even hard.
> >
> > Ok, so this isn't a hazy unattainable dream, perfect!
> >
> >
> > > unbind cards or unload the module and RX 480
> >
> > The host would get the IGP from the CPU. Both PCIE cards would be
> > vfio-stubed, so as far as I understand it these cards will be handled by
> > the vfio module in the host kernel and should be freely assignable to
> > guests. Therefore don't use any power if no guest is using it.
>
> Um, not quite.  Unless you have hotplug capable slots with independent
> power control, the best we can do is put devices into a "low power", D3
> power state.  The PCI spec requires devices to support this state, but
> does not require devices to meet any particular standards for power
> consumption in this state.  I would characterize D3 as slightly better
> than doing nothing, no more.  In fact, the power consumption might be
> lower with the VM running and the guest driver managing the device
> specific power controls.
>
> > I'm not deadset on Nvidia I rather would use something else because of
> the
> > uncertainty that comes with and Nvidia driver. My guess is that their
> > proprietary Nvidia linux driver would have VM detection too, but didn't
> > find confirmation on this. I would like to use a AMD graphics card just
> to
> > be safe and out of that armsrace, AMDs driver doesn't block like Nvidia
> > does.
> > But I got discouraged by AMD graphics card PCIE reset issue reports and I
> > couldn't make heads and tails about it. If any RX 480 works, great. I
> will
> > have to look more into this.
> >
> >
> > > My Linux desktop get the IGD [..] Why not make more use of IGD
> assignment
> > > in your configuration.
> >
> > Does a host system even boot with all graphics devices stubed? So am I
> > right to assuming that x86 "graphic needs" are satisfied for POST but
> after
> > that doesn't care what happens to the graphics devices afterwards? I
> don't
> > mind a headless host at all, it will change somewhat how I setup things
> for
> > crypto/storage but that I can solve.
>
> Linux doesn't require a graphics head, I use a serial console for my
> system.  I personally don't like entirely headless configurations (ie.
> no graphical or serial console), but to each their own.
>
> > For the IGP to be stubed and passed through I need a ACS capable CPU? As
> it
> > seems I wrongfully assumed that Vt-d and Vt-x support entailed this.
>
> IGD is not dependent on ACS, IGD is a single function device connected
> directly to the PCIe root complex.  It will always be in an IOMMU group
> of its own.  Unfortunately the Intel systems that include proper ACS do
> not include IGD, and conversely, if you have IGD you do not have ACS.
> Plan your usage accordingly.
>
> > > Concerns with your hardware choice otherwise: a) No ACS, you won't be
> > able
> > > to assign cards in CPU root port slots to separate VMs
> > > (see vfio.blogspot.com), b) insufficient cores, how much are you
> > > willing to have your storage server interfere with your gaming
> > performance?
> >
> > a) Ok, I found your article http://vfio.blogspot.de/2015/
> > 10/intel-processors-with-acs-support.html these are the only choices?
>
> At the time of writing, I think it was fairly comprehensive, since
> Intels product SKU marketing scramble with "Scalable Processors" I'm
> not sure we've figured out the new decoder ring yet.
>
> > Doesn't AMD support this too? AMD CPUs are usually weaker in single core
> > performance, so I ignored AMD CPUs for years. But seeing the prices I
> > wonder if they have an alternative for ~450 Euro price range. Some of the
> > Intel CPUs you listed are way out of my budget. The strongest and
> somewhat
> > affordable listed is a i7-6850k. AMD ryzen 1800x costs ~470 Euro, but
> > doesn't have an IGP and I didn't look into AMD CPUs at all so I don't
> know
> > if they support ACS it has AMD-V tho.
>
> Previous generation, 990-based AM3/+ sockets supported ACS.  Ryzen
> systems seem to have implemented it in hardware, but nobody told the
> BIOS team that it was important to enable and support for ACS is slowly
> being added.  Specifications are hard to come by for AMD chips though,
> so we only have user reports to tell us how various systems fair.
> Increased core count in Ryzen is certainly desirable for virtualization
> scenarios, but the number of PCIe lanes from the processor can lead to
> limited I/O options or additional switch components that may ruin the
> ACS provided by the root ports.  Note that there's also an assigned GPU
> performance issue with AMD's NPT as well that isn't understood.
>
> > b) Well. I'm used to Intel quad core CPUs. Seeing that no matter what I
> buy
> > I will endup with 6 cores at least, I guess I can live with it. Desktop
> > guest and host heavyload-PIDs will get taskset to one core and the gaming
> > guest gets the rest pinned and taskset. So my guess is that any
> > crypt/storage/desktop spike shouldn't be noticed in the gaming guest.
>
> 6-cores?  At best you can get 4-cores + threads on Intel processors
> with IGD.  Folks here tend to complain about every little micro jitter
> they see in VM configurations and those can be difficult to eliminate
> entirely without cores that can be dedicated to the gaming VM.
>
> > > The only major issue is that running things involving sound and video
> > becomes
> > > not possible on such desktop. But those can be viewed/listened
> locally, on
> > > whatever actual machine you happen to be at. (Or also there are ways to
> > pass
> > > through sound from a remote desktop, or use video-efficient protocols
> > such as
> > > SPICE, but for my use I did not look into those as of yet).
> >
> > This would be an issue. I didn't look into audio and microphone that
> much.
> > I would like to hear the audio from desktop guest and gaming guest and
> have
> > microphone available at least in the gaming guest for voice chat. Right
> now
> > this doesn't matter that much. Host and guests will be all GNU/Linux
> > distributions so without having looked into it further I guess there is a
> > server solution if any audio/mic passthrough scenario wouldn't work. I
> > guess your SPICE suggestion is such a solution.
> > Monitor wise I initially thought about HDMI switching host and desktop
> > guest, but as it seems that I can vfio-stub all graphics devices with a
> > proper CPU, I won't need the passive PCIE card in the case of i7-6850k
> > anymore and can use the IGP for the desktop guest instead. The desktop
> > guest will be dual monitor setup and the gaming guest will be switched on
> > one of the monitors, like I have it right now with separate computers.
> > Keyboard + mouse, I already use a two port usb switch. I guess I can
> still
> > use that to switch between desktop and gaming guests, after one of the
> two
> > usb-switch cables is passed through to the desktop guest after boot.
> >
> > I'm exited and a bit confused.
> >
> > How do you see if a CPU does support proper isolation or not? It isn't
> the
> > feature list like AMD-V or Vt-d/x only. Is it a combination of mainboard
> > chipset and CPU? I ask because I see the TDP wattage difference between
> > ryzen 7 1800x and the i7-6850k. So being relatively equal in price and
> > performance I tend towards the ryzen CPU. In the last couple hours I fail
> > to find if it would support proper PCIE isolation or not. In the search
> > stumbled upon ACS kernel patch, if possible I don't want to apply the ACS
> > patch to the host kernel.
>
> The ACS override patch is unsupported and not destined for upstream, so
> I would certainly not use it.  ACS is typically included in the
> register definition in the vol2 Intel datasheet.  For AMD, good luck,
> we rely on user reports due to lack of published specs.  We know
> that AGESA 1.0.0.6 is required for Ryzen root port ACS and I see
> on another list that 1.0.0.7 is claimed will enable ACS on downstream
> switch ports for an internal switch in X370.  If we can get to the
> bottom of the NPT performance issue, Ryzen may yet shape up to be a
> good device assignment platform, but it's taking its time to get
> there.  Thanks,
>
> Alex
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20170717/21a97fd6/attachment.htm>