[vfio-users] Newbie steps

[Torbjorn Jansson]

On 2017-03-11 14:24, Patrick O'Callaghan wrote:
> I'm a newbie when it comes to this stuff so would appreciate some
> guidance. I've read Alex's long blog post and several other sources but
> am still having trouble at the most basic level, i.e. masking my Nvidia
> card.
>
> The system is an i7-3770 running Fedora 25 on an MSI board. VT-d is
> enabled (as confirmed from dmesg). The second GPU is an Geforce GT630
> (fairly low-end but I'll swap in a better one if I can get this all to
> work). Basic info:
>
> $ lspci
> 00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor DRAM Controller (rev 09)
> 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)
> 00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller (rev 04)
> 00:16.0 Communication controller: Intel Corporation 7 Series/C216 Chipset Family MEI Controller #1 (rev 04)
> 00:1a.0 USB controller: Intel Corporation 7 Series/C216 Chipset Family USB Enhanced Host Controller #2 (rev 04)
> 00:1b.0 Audio device: Intel Corporation 7 Series/C216 Chipset Family High Definition Audio Controller (rev 04)
> 00:1c.0 PCI bridge: Intel Corporation 7 Series/C216 Chipset Family PCI Express Root Port 1 (rev c4)
> 00:1c.1 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 2 (rev c4)
> 00:1d.0 USB controller: Intel Corporation 7 Series/C216 Chipset Family USB Enhanced Host Controller #1 (rev 04)
> 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a4)
> 00:1f.0 ISA bridge: Intel Corporation B75 Express Chipset LPC Controller (rev 04)
> 00:1f.2 SATA controller: Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode] (rev 04)
> 00:1f.3 SMBus: Intel Corporation 7 Series/C216 Chipset Family SMBus Controller (rev 04)
> 01:00.0 VGA compatible controller: NVIDIA Corporation GF108 [GeForce GT 630] (rev a1)
> 01:00.1 Audio device: NVIDIA Corporation GF108 High Definition Audio Controller (rev a1)
> 03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)
>
> $ find /sys/kernel/iommu_groups/ -type l
> /sys/kernel/iommu_groups/7/devices/0000:00:1c.1
> /sys/kernel/iommu_groups/5/devices/0000:00:1b.0
> /sys/kernel/iommu_groups/3/devices/0000:00:16.0
> /sys/kernel/iommu_groups/11/devices/0000:03:00.0
> /sys/kernel/iommu_groups/1/devices/0000:01:00.1
> /sys/kernel/iommu_groups/1/devices/0000:00:01.0
> /sys/kernel/iommu_groups/1/devices/0000:01:00.0
> /sys/kernel/iommu_groups/8/devices/0000:00:1d.0
> /sys/kernel/iommu_groups/6/devices/0000:00:1c.0
> /sys/kernel/iommu_groups/4/devices/0000:00:1a.0
> /sys/kernel/iommu_groups/2/devices/0000:00:14.0
> /sys/kernel/iommu_groups/10/devices/0000:00:1f.3
> /sys/kernel/iommu_groups/10/devices/0000:00:1f.2
> /sys/kernel/iommu_groups/10/devices/0000:00:1f.0
> /sys/kernel/iommu_groups/0/devices/0000:00:00.0
> /sys/kernel/iommu_groups/9/devices/0000:00:1e.0
>
> So IOMMU group 1 contains my Nvidia GPU and audio, plus the PCIe root
> bridge.
>
> $ cat /etc/modprobe.d/local.conf
> options vfio-pci ids=10de:ffffffff:ffffffff:ffffffff:00030000:ffff00ff,10de:ffffffff:ffffffff:ffffffff:00040300:ffffffff
>
> (this is just cut-and-paste from Alex's blog. I've no idea what any of
> the magic hex numbers mean except for the 10de of course).
>
> $ cat /etc/dracut.conf.d/local.conf
> add_drivers+="vfio vfio_iommu_type1 vfio_pci vfio_virqfd"
>
> My boot line is:
>
> GRUB_CMDLINE_LINUX="rd.blacklist=nvidia rd.blacklist=nouveau intel_iommu=on iommu=pt rd.driver.pre=vfio-pci vconsole.font=latarcyrheb-sun16 $([ -x /usr/sbin/rhcrashkernel-param ] && /usr/sbin/rhcrashkernel-param || :) rhgb quiet resume=U
> UID=1431e6d2-531e-46cd-8633-1cf878c6b2a1" audit=0
>
> (Note that I have the Nvidia proprietary driver so need to mask that as
> well as Nouveau).
>
> My monitor is connected to HDMI on the internal GPU and DVI on the
> Nvidia card.
>
> However on rebooting I always have the Nvidia module loaded, and don't
> have vfio:
> $ lsmod|egrep -i nvidia\|vfio
> nvidia_drm             53248  1
> nvidia_modeset        806912  8 nvidia_drm
> nvidia              12267520  175 nvidia_modeset
> drm_kms_helper        151552  1 nvidia_drm
> drm                   339968  4 nvidia_drm,drm_kms_helper
>
> And of course, the monitor is only active on the DVI port and not the HDMI.
>
> Any suggestions on what I might try?
>
> poc
>

looks ok, did you remember to regenerate grub config?
if not, then your changes in /etc/default/grub did not take effect.

if i'm not mistaken you run: grub2-mkconfig -o grub.cfg
check it first before replacing old file under /boot and make a backup 
of it first in case you screw up.