[vfio-users] [FEEDBACK NEEDED] Additionnal tips to include in the Arch wiki article

[Bronek Kozicki]

On 20/05/2016 18:38, Nicolas Roy-Renaud wrote:
> Did you have any issues with efifb or some other driver grabbing the
> gard before vfio-pci? What are your IOMMU groups like, which chard is
> your boot_vga, what script or method do you use to get vfio-pci to bind
> with your card, what's your kernel command line and what modules are
> included in your initramfs?
>
> I've struggled with getting my primary GPU to work correctly for a
> passthrough like this, and I ended up switching it up with the secondary
> because I simply couldn't get it to work properly (I'd keep getting
> something about "|Invalid ROM contents|") and I couldn't see my boot
> logs until the host driver finished loading. It's really inconvenient
> now becuase my motherboard is designed so that having a full-size GPU in
> the second PCIe slot blocks 4 SATA ports out of 6, so I need to have
> L-shaped cables in a bunch of places to make everything work properly.


I am starting from primary GPU. More specifically I have two GPUs, both 
for vfio passthrough for different guests, and the host remains 
headless. My guests both start with OVMF , but setup used to work also 
with legacy VGA. Both GPUs are setup in BIOS to initialise their ROM, 
but virtual machine start *without* ROM, i.e. I am passing <rom 
bar='off'/> option in libvirt device definition (which is translated to 
qemu "-device vfio-pci,host=....,rombar=0") and as a result I do not see 
guest boot screen, but that's fine for me.

I also have a set of "belt-and-braces" options to prevent host from 
using GPUs, these are:

% cat /etc/modprobe.d/vfio.conf
options vfio-pci ids=1912:0014,10de:13f0,10de:0fbb
options vfio-pci disable_vga=1

% cat /proc/cmdline
BOOT_IMAGE=../vmlinuz-linux console=ttyS0,115200N8R nomodest 
video=vesa:off video=efifb:off vga=normal nohz=off intel_iommu=on iommu=pt
(removed other irrelevant options)

% cat /etc/modprobe.d/blacklist.conf
# This host is headless, prevent any modules from attaching to video 
hardware
# NVIDIA
blacklist nouveau
blacklist nvidia
# AMD
blacklist radeon
blacklist amdgpu
blacklist amdkfd
blacklist fglrx
# HDMI sound on a GPU
blacklist snd_hda_intel
# Framebuffers (ALL of them)
blacklist vesafb
blacklist aty128fb
blacklist atyfb
blacklist radeonfb
blacklist cirrusfb
blacklist cyber2000fb
blacklist cyblafb
blacklist gx1fb
blacklist hgafb
blacklist i810fb
blacklist intelfb
blacklist kyrofb
blacklist lxfb
blacklist matroxfb_base
blacklist neofb
blacklist nvidiafb
blacklist pm2fb
blacklist rivafb
blacklist s1d13xxxfb
blacklist savagefb
blacklist sisfb
blacklist sstfb
blacklist tdfxfb
blacklist tridentfb
blacklist vfb
blacklist viafb
blacklist vt8623fb
blacklist udlfb

Also my syslinux configuration starts with
SERIAL 0 115200 0xab3

I am quite sure that Grub allows for similar option to use serial port 
instead of VGA.

If you do not want your host headless, I think your options are:
* swap GPUs around
* disable ROM on slot where primary GPU is attached, thus "downgrading" 
it to secondary role
* let vfio manage transfer of GPU from host to guest (hopefully 
supported by open source drivers, but you may need to blacklist closed 
source ones). Requires guest to use OVMF, I think.


B.