[vfio-users] IGD pass through, black screen, block on running option rom in seabios

Thu Aug 23 19:32:39 UTC 2018

On Thu, 23 Aug 2018 15:57:11 +0800
acewind <acewind at gmail.com> wrote:

> Yesterday I tried IGD with Ubuntu 18.04 but failed:
> https://www.redhat.com/archives/vfio-users/2018-August/msg00026.html
> 
> Today I use the same hardware enviroment, reinstall OS to CentOS 7.3
> without GUI, upgrade kernel and qemu version:
> 
> CPU: i3-4010U, support vt-d.
> 
> [root at test ~]# lspci | grep VGA
> 00:02.0 VGA compatible controller: Intel Corporation Haswell-ULT Integrated
> Graphics Controller (rev 09)
> (Only one IGD with vga and hdmi interface)
> 
> [root at test ~]# uname -r
> 4.18.3-1.el7.centos.x86_64
> 
> [root at test ~]# /usr/libexec/qemu-kvm --version
> QEMU emulator version 2.10.0(qemu-kvm-ev-2.10.0-21.el7_5.4.1)
> Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers

You might be going further off track with this, the official RH
products do not support IGD assignment in any way other than kvmgt, ie.
GVT-g, ie. vGPU.  In fact the RHEL kernel specifically disables some of
the IGD and VGA features necessary for legacy mode IGD assignment.  I
don't know how much of this Centos carries to their product, but if
you're moving to Centos with the expectation that IGD assignment is
better supported there from the RH lineage, it's not.

> Then I blacklist i915, forbid *fb:
> 
> [root at test ~]# cat /etc/default/grub | grep GRUB_CMDLINE_LINUX
> GRUB_CMDLINE_LINUX="crashkernel=auto rhgb quiet intel_iommu=on iommu=pt
> video=vesafb:off vga=normal nofb nomodeset i915.modeset=0
> nouveau.modeset=0 rd.driver.blacklist=nouveau,i915 plymouth.ignore-udev"
> 
> [root at test ~]# cat /etc/modprobe.d/blacklist-i915.conf
> install i915 /bin/true
> 
> After rebooted:
> [root at test ~]# cat /proc/cmdline
> BOOT_IMAGE=/vmlinuz-4.18.3-1.el7.centos.x86_64
> root=UUID=78feccee-2f23-4ab6-b386-2b51d1664da7 ro crashkernel=auto rhgb
> quiet intel_iommu=on iommu=pt video=vesafb:off vga=normal nofb nomodeset
> i915.modeset=0 nouveau.modeset=0 rd.driver.blacklist=nouveau,i915
> plymouth.ignore-udev
> 
> [root at test ~]# cat /proc/iomem | grep fb
>   01c031d1-02463fbf : Kernel data
> f8000000-fbffffff : PCI MMCONFIG 0000 [bus 00-3f]
>   f8000000-fbffffff : Reserved
>     f8000000-fbffffff : pnp 00:05
> 
> [root at test ~]# lsmod | grep i915
> ... nothing ...

It's really not necessary to go to these sorts of lengths to disable
i915.  You can simply 'virsh nodedev-detach pci_0000_00_02_0'.  IME,
the place where i915 causes trouble is if you later attempt to re-bind
the device to i915.

> Now I begin to load vfio modules.
> 
> [root at test ~]# modprobe vfio
> [root at test ~]# modprobe vfio-pci
> 
> [root at test ~]# echo "vfio-pci" >
> /sys/bus/pci/devices/0000:00:02.0/driver_override
> [root at test ~]# ./vfio-pci-bind.sh 0000:00:02.0
> (script ref:
> https://github.com/andre-richter/vfio-pci-bind/blob/master/vfio-pci-bind.sh)
> 
> [root at test ~]# lspci -v -s 00:02.0
> 00:02.0 VGA compatible controller: Intel Corporation Haswell-ULT Integrated
> Graphics Controller (rev 09) (prog-if 00 [VGA controller])
>     Subsystem: Intel Corporation Haswell-ULT Integrated Graphics Controller
>     Flags: fast devsel, IRQ 16
>     Memory at a7c00000 (64-bit, non-prefetchable) [size=4M]
>     Memory at b0000000 (64-bit, prefetchable) [size=256M]
>     I/O ports at 4000 [size=64]
>     [virtual] Expansion ROM at 000c0000 [disabled] [size=128K]
>     Capabilities: [90] MSI: Enable- Count=1/1 Maskable- 64bit-
>     Capabilities: [d0] Power Management version 2
>     Capabilities: [a4] PCI Advanced Features
>     Kernel driver in use: vfio-pci
>     Kernel modules: i915
> 
> Now virsh create a win7 vm, xml file is:
> 
> <domain type='kvm' id='3' xmlns:qemu='
> http://libvirt.org/schemas/domain/qemu/1.0'>  
>   <name>windows_7_ultimate_with_sp1_x64</name>
>   <uuid>2fcccb90-f622-481a-8819-367c642c0a09</uuid>
>   <memory unit='KiB'>2097152</memory>
>   <currentMemory unit='KiB'>2097152</currentMemory>
>   <vcpu placement='static'>2</vcpu>
>   <resource>
>     <partition>/machine</partition>
>   </resource>
>   <os>
>     <type arch='x86_64' machine='pc-i440fx-rhel7.5.0'>hvm</type>
>     <boot dev='hd'/>
>   </os>
>   <features>
>     <acpi/>
>     <apic/>
>     <hyperv>
>       <relaxed state='on'/>
>       <vapic state='on'/>
>       <spinlocks state='on' retries='8191'/>
>     </hyperv>
>   </features>
>   <cpu mode='custom' match='exact' check='full'>
>     <model fallback='forbid'>Haswell-noTSX</model>
>     <feature policy='require' name='vme'/>
>     <feature policy='require' name='f16c'/>
>     <feature policy='require' name='rdrand'/>
>     <feature policy='require' name='hypervisor'/>
>     <feature policy='require' name='arat'/>
>     <feature policy='require' name='xsaveopt'/>
>     <feature policy='require' name='abm'/>
>   </cpu>
>   <clock offset='localtime'>
>     <timer name='rtc' tickpolicy='catchup'/>
>     <timer name='pit' tickpolicy='delay'/>
>     <timer name='hpet' present='no'/>
>     <timer name='hypervclock' present='yes'/>
>   </clock>
>   <on_poweroff>destroy</on_poweroff>
>   <on_reboot>restart</on_reboot>
>   <on_crash>destroy</on_crash>
>   <pm>
>     <suspend-to-mem enabled='no'/>
>     <suspend-to-disk enabled='no'/>
>   </pm>
>   <devices>
>     <emulator>/usr/libexec/qemu-kvm</emulator>
>     <disk type='file' device='disk'>
>       <driver name='qemu' type='qcow2'/>
>       <source file='/home/windows_7_ultimate_with_sp1_x64.qcow2'/>
>       <backingStore/>
>       <target dev='hda' bus='ide'/>
>       <alias name='ide0-0-0'/>
>       <address type='drive' controller='0' bus='0' target='0' unit='0'/>
>     </disk>
> 
>     <hostdev mode='subsystem' type='pci' managed='yes'>
>       <driver name='vfio'/>
>       <source>
>         <address domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
>       </source>
>       <alias name='hostdev0'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> function='0x0'/>  
>     </hostdev>
>   </devices>
> 
>     <qemu:commandline>
>      <qemu:arg value='-bios'/>
>       <qemu:arg value='/usr/share/seabios/bios.bin'/>
>       <qemu:arg value='-chardev'/>
>       <qemu:arg value='file,id=seabios,path=/tmp/bios.log'/>
>       <qemu:arg value='-device'/>
>       <qemu:arg value='isa-debugcon,iobase=0x402,chardev=seabios'/>
>     </qemu:commandline>
> </domain>
> 
> After created, the vm is running, monitor screen turn to black then no
> output.
> The end of seabios log is:
> 
> ...
> Scan for VGA option rom
> Running option rom at c000:0003
> 
> Any problem with VGA rom? In the BIOS I have already enabled CSM and the
> option rom of video is set to legacy.
> 
> Ref to http://lists.gnu.org/archive/html/qemu-discuss/2018-04/msg00052.html
> and https://github.com/awilliam/rom-parser
> 
> I dump the vga rom:
> 
> [root at test rom]# echo 1 > /sys/devices/pci0000:00/0000:00:02.0/rom
> [root at test rom]# cat /sys/devices/pci0000:00/0000:00:02.0/rom > vbios.dump
> [root at test rom]# echo 0 > /sys/devices/pci0000:00/0000:00:02.0/rom
> 
> [root at test rom]# ./rom-parser/rom-parser vbios.dump
> Valid ROM signature found @0h, PCIR offset 40h
>     PCIR: type 0 (x86 PC-AT), vendor: 8086, device: 0406, class: 030000
>     PCIR: revision 3, vendor revision: 0
>     Last image
> 
> [root at test rom]# lspci -nns 00:02.0
> 00:02.0 VGA compatible controller [0300]: Intel Corporation Haswell-ULT
> Integrated Graphics Controller [8086:0a16] (rev 09)
> 
> The device code in the rom is diffrent from vga device? So I modify it with
> rom-fixer command:
> 
> [root at test rom]# ./rom-parser/rom-fixer ./vbios.dump
> Valid ROM signature found @0h, PCIR offset 40h
>     PCIR: type 0 (x86 PC-AT), vendor: 8086, device: 0406, class: 030000
>     PCIR: revision 3, vendor revision: 0
> 
> Modify vendor ID 8086? (y/n): y
> New vendor ID: 8086
> Overwrite vendor ID with 8086? (y/n): y
> Modify device ID 0406? (y/n): y
> New device ID: 0a16
> Overwrite device ID with 0a16? (y/n): y
>     Last image
> ROM checksum is invalid, fix? (y/n): y

The only reason this should be necessary is if you're already using an
externally provided rom via <rom>.  QEMU will do the above for you
automatically when it's reading the rom directly from the device.  So
there's no net effect of this change.

> Then I edit the vm xml, and romfile args to hostdev:
>       <rom file='/path/to/vbios.dump'/>
> 
> Then destroy and create the vm again, still black screen ...
> 
> Is there any more methods to debug？

I'd use upstream kernel and QEMU.  Your previous post indicated DMAR
faults.  A couple of those are normal, but you seem to have a fair
number of them and the address is incrementing as if our attempt to
trap and relocate stolen memory isn't working.  It's possible this
particular GPU uses a different page table format that we don't know
how to handle yet.  Thanks,

Alex