[vfio-users] Newbie steps

[Patrick O'Callaghan]

I'm a newbie when it comes to this stuff so would appreciate some
guidance. I've read Alex's long blog post and several other sources but
am still having trouble at the most basic level, i.e. masking my Nvidia
card.

The system is an i7-3770 running Fedora 25 on an MSI board. VT-d is
enabled (as confirmed from dmesg). The second GPU is an Geforce GT630
(fairly low-end but I'll swap in a better one if I can get this all to
work). Basic info:

$ lspci
00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor DRAM Controller (rev 09)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)
00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller (rev 04)
00:16.0 Communication controller: Intel Corporation 7 Series/C216 Chipset Family MEI Controller #1 (rev 04)
00:1a.0 USB controller: Intel Corporation 7 Series/C216 Chipset Family USB Enhanced Host Controller #2 (rev 04)
00:1b.0 Audio device: Intel Corporation 7 Series/C216 Chipset Family High Definition Audio Controller (rev 04)
00:1c.0 PCI bridge: Intel Corporation 7 Series/C216 Chipset Family PCI Express Root Port 1 (rev c4)
00:1c.1 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 2 (rev c4)
00:1d.0 USB controller: Intel Corporation 7 Series/C216 Chipset Family USB Enhanced Host Controller #1 (rev 04)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a4)
00:1f.0 ISA bridge: Intel Corporation B75 Express Chipset LPC Controller (rev 04)
00:1f.2 SATA controller: Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode] (rev 04)
00:1f.3 SMBus: Intel Corporation 7 Series/C216 Chipset Family SMBus Controller (rev 04)
01:00.0 VGA compatible controller: NVIDIA Corporation GF108 [GeForce GT 630] (rev a1)
01:00.1 Audio device: NVIDIA Corporation GF108 High Definition Audio Controller (rev a1)
03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)

$ find /sys/kernel/iommu_groups/ -type l
/sys/kernel/iommu_groups/7/devices/0000:00:1c.1
/sys/kernel/iommu_groups/5/devices/0000:00:1b.0
/sys/kernel/iommu_groups/3/devices/0000:00:16.0
/sys/kernel/iommu_groups/11/devices/0000:03:00.0
/sys/kernel/iommu_groups/1/devices/0000:01:00.1
/sys/kernel/iommu_groups/1/devices/0000:00:01.0
/sys/kernel/iommu_groups/1/devices/0000:01:00.0
/sys/kernel/iommu_groups/8/devices/0000:00:1d.0
/sys/kernel/iommu_groups/6/devices/0000:00:1c.0
/sys/kernel/iommu_groups/4/devices/0000:00:1a.0
/sys/kernel/iommu_groups/2/devices/0000:00:14.0
/sys/kernel/iommu_groups/10/devices/0000:00:1f.3
/sys/kernel/iommu_groups/10/devices/0000:00:1f.2
/sys/kernel/iommu_groups/10/devices/0000:00:1f.0
/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/9/devices/0000:00:1e.0

So IOMMU group 1 contains my Nvidia GPU and audio, plus the PCIe root
bridge. 

$ cat /etc/modprobe.d/local.conf  
options vfio-pci ids=10de:ffffffff:ffffffff:ffffffff:00030000:ffff00ff,10de:ffffffff:ffffffff:ffffffff:00040300:ffffffff

(this is just cut-and-paste from Alex's blog. I've no idea what any of
the magic hex numbers mean except for the 10de of course).

$ cat /etc/dracut.conf.d/local.conf
add_drivers+="vfio vfio_iommu_type1 vfio_pci vfio_virqfd"

My boot line is:

GRUB_CMDLINE_LINUX="rd.blacklist=nvidia rd.blacklist=nouveau intel_iommu=on iommu=pt rd.driver.pre=vfio-pci vconsole.font=latarcyrheb-sun16 $([ -x /usr/sbin/rhcrashkernel-param ] && /usr/sbin/rhcrashkernel-param || :) rhgb quiet resume=U
UID=1431e6d2-531e-46cd-8633-1cf878c6b2a1" audit=0

(Note that I have the Nvidia proprietary driver so need to mask that as
well as Nouveau).

My monitor is connected to HDMI on the internal GPU and DVI on the
Nvidia card.

However on rebooting I always have the Nvidia module loaded, and don't
have vfio:
$ lsmod|egrep -i nvidia\|vfio
nvidia_drm             53248  1
nvidia_modeset        806912  8 nvidia_drm
nvidia              12267520  175 nvidia_modeset
drm_kms_helper        151552  1 nvidia_drm
drm                   339968  4 nvidia_drm,drm_kms_helper

And of course, the monitor is only active on the DVI port and not the HDMI.

Any suggestions on what I might try?

poc