[vfio-users] Questions for all who have gotten this to work

[Alex Williamson]

On Thu, 2015-09-03 at 16:10 +0200, Zycorax Tokoroa wrote:
> I hope no one minds if I add my own question at this. I'd like to hear
> suggestions on the second GPU to take, keeping an eye on the power drain
> and the heat generated.
> 
> The current hardware configuration is this:
> Corsair Cx750M
> ASUS X99-Deluxe (which seems to support ACS and hence need no patch?)

Depends on where you install the card.  We have quirks for the X99
chipset (and in fact most Intel chipsets, except the latest) to enforce
isolation of the PCH root ports, but Intel client processors (Core
i7/i5) and Xeon E3 do not support ACS on the processor root ports.
You'd need a Xeon E5 platform to be completely clear of the ACS issues.

> ASUS STRIX GTX 970
> Intel i7-5930K
> 
> I take that it's recommended, in order to have other configurations
> available, such as both GPUs working in parallel on the same (host) OS,
> to avoid using both NVidia and ATI, and the graphic cards have to be the
> same model (although different OCs are theoretically allowed) in case of
> SLI.
> 
> I suppose that this should restrict the best choice between another 970
> and a 960

Generally it's recommended that a GPU for a VM is dedicated to the VM,
graphics drivers on Linux don't really like unbinding devices.  The
Nvidia proprietary driver in particular will not cleanly unbind from a
device to assign to a guest and a bug filed against Nvidia for this
issue was closed wontfix.  That doesn't prevent rebooting between
configurations where the host can take advantage of multiple GPUs, but
AFAIK nobody has gotten SLI to work in a VM, so strike that out as a
possibility unless someone is willing to invest development time on it.

All things considered, I personally prefer Nvidia for VMs, even despite
their passive aggressive actions to prevent GeForce assignment.  Their
cards handle reset and after you get past the Code 43 issues (and
assuming they don't add new ones), the drivers don't BSOD like the AMD
ones are prone to.  Maxwell has quite nice power efficiency as well.

If you choose the right AMD card, they can work well and I'm sure others
will chime in with their success stories for AMD.

And to answer the original questions:

> What distro do you use?

Fedora

> What kernel are you running? Did you have to re-compile it?

4.1.x, stock distro

> What CPU and GPUs do you use?

Xeon E3-1245v2 + EVGA GTX750 SC.  I also have an HD8570 configuration in
the same box as documented in my blog.

> What online tutorial did you use?

My own :)

> How long have you had it going in a stable fashion?

Running (with different cards) since before my KVM Forum 2013
presentation, so roughly two years.  I'd only consider it stable since
being able to use stock QEMU and kernel with OVMF, so more like one
year.