[vfio-users] Newbie steps

[Alex Williamson]

On Sat, Mar 11, 2017 at 6:24 AM, Patrick O'Callaghan <poc at usb.ve> wrote:

> I'm a newbie when it comes to this stuff so would appreciate some
> guidance. I've read Alex's long blog post and several other sources but
> am still having trouble at the most basic level, i.e. masking my Nvidia
> card.
>
> The system is an i7-3770 running Fedora 25 on an MSI board. VT-d is
> enabled (as confirmed from dmesg). The second GPU is an Geforce GT630
> (fairly low-end but I'll swap in a better one if I can get this all to
> work). Basic info:
>
> $ lspci
> 00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core
> processor DRAM Controller (rev 09)
> 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core
> processor PCI Express Root Port (rev 09)
>

There's no 00:02.0 VGA device here, which means IGD is disabled and only
the NVIDIA card is being exposed to the OS.  Assigning primary graphics to
the guest is generally not recommended unless you know how to work around
the labyrinth of issues that may result (ie. avoid if a newbie).  Enable
IGD and set it as the primary host graphics.


> 00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset
> Family USB xHCI Host Controller (rev 04)
> 00:16.0 Communication controller: Intel Corporation 7 Series/C216 Chipset
> Family MEI Controller #1 (rev 04)
> 00:1a.0 USB controller: Intel Corporation 7 Series/C216 Chipset Family USB
> Enhanced Host Controller #2 (rev 04)
> 00:1b.0 Audio device: Intel Corporation 7 Series/C216 Chipset Family High
> Definition Audio Controller (rev 04)
> 00:1c.0 PCI bridge: Intel Corporation 7 Series/C216 Chipset Family PCI
> Express Root Port 1 (rev c4)
> 00:1c.1 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family
> PCI Express Root Port 2 (rev c4)
> 00:1d.0 USB controller: Intel Corporation 7 Series/C216 Chipset Family USB
> Enhanced Host Controller #1 (rev 04)
> 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a4)
> 00:1f.0 ISA bridge: Intel Corporation B75 Express Chipset LPC Controller
> (rev 04)
> 00:1f.2 SATA controller: Intel Corporation 7 Series/C210 Series Chipset
> Family 6-port SATA Controller [AHCI mode] (rev 04)
> 00:1f.3 SMBus: Intel Corporation 7 Series/C216 Chipset Family SMBus
> Controller (rev 04)
> 01:00.0 VGA compatible controller: NVIDIA Corporation GF108 [GeForce GT
> 630] (rev a1)
> 01:00.1 Audio device: NVIDIA Corporation GF108 High Definition Audio
> Controller (rev a1)
> 03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd.
> RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)
>
> $ find /sys/kernel/iommu_groups/ -type l
> /sys/kernel/iommu_groups/7/devices/0000:00:1c.1
> /sys/kernel/iommu_groups/5/devices/0000:00:1b.0
> /sys/kernel/iommu_groups/3/devices/0000:00:16.0
> /sys/kernel/iommu_groups/11/devices/0000:03:00.0
> /sys/kernel/iommu_groups/1/devices/0000:01:00.1
> /sys/kernel/iommu_groups/1/devices/0000:00:01.0
> /sys/kernel/iommu_groups/1/devices/0000:01:00.0
> /sys/kernel/iommu_groups/8/devices/0000:00:1d.0
> /sys/kernel/iommu_groups/6/devices/0000:00:1c.0
> /sys/kernel/iommu_groups/4/devices/0000:00:1a.0
> /sys/kernel/iommu_groups/2/devices/0000:00:14.0
> /sys/kernel/iommu_groups/10/devices/0000:00:1f.3
> /sys/kernel/iommu_groups/10/devices/0000:00:1f.2
> /sys/kernel/iommu_groups/10/devices/0000:00:1f.0
> /sys/kernel/iommu_groups/0/devices/0000:00:00.0
> /sys/kernel/iommu_groups/9/devices/0000:00:1e.0
>
> So IOMMU group 1 contains my Nvidia GPU and audio, plus the PCIe root
> bridge.
>
> $ cat /etc/modprobe.d/local.conf
> options vfio-pci ids=10de:ffffffff:ffffffff:ffffffff:00030000:ffff00ff,
> 10de:ffffffff:ffffffff:ffffffff:00040300:ffffffff
>
> (this is just cut-and-paste from Alex's blog. I've no idea what any of
> the magic hex numbers mean except for the 10de of course).
>
> $ cat /etc/dracut.conf.d/local.conf
> add_drivers+="vfio vfio_iommu_type1 vfio_pci vfio_virqfd"
>
> My boot line is:
>
> GRUB_CMDLINE_LINUX="rd.blacklist=nvidia rd.blacklist=nouveau
> intel_iommu=on iommu=pt rd.driver.pre=vfio-pci vconsole.font=latarcyrheb-sun16
> $([ -x /usr/sbin/rhcrashkernel-param ] && /usr/sbin/rhcrashkernel-param ||
> :) rhgb quiet resume=U
> UID=1431e6d2-531e-46cd-8633-1cf878c6b2a1" audit=0
>
> (Note that I have the Nvidia proprietary driver so need to mask that as
> well as Nouveau).
>

Or just uninstall it.


> My monitor is connected to HDMI on the internal GPU and DVI on the
> Nvidia card.
>

IGD is disabled in BIOS according to the info presented.


> However on rebooting I always have the Nvidia module loaded, and don't
> have vfio:
> $ lsmod|egrep -i nvidia\|vfio
> nvidia_drm             53248  1
> nvidia_modeset        806912  8 nvidia_drm
> nvidia              12267520  175 nvidia_modeset
> drm_kms_helper        151552  1 nvidia_drm
> drm                   339968  4 nvidia_drm,drm_kms_helper
>

What do you actually see for the commandline in /proc/cmdline?


>
> And of course, the monitor is only active on the DVI port and not the HDMI.
>
> Any suggestions on what I might try?
>

Enable IGD in the BIOS, set it as primary graphics, uninstall nvidia driver
in the host, verify the kernel commandline is what you think via proc and
that your initramfs is being rebuilt with the drivers you want, use
lsinitrd to check.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20170312/bb688558/attachment.htm>