[vfio-users] IGD pass through, black screen, block on running option rom in seabios
Acewind
acewind at gmail.com
Fri Aug 24 12:23:17 UTC 2018
Now I try Fedora 25 on another enviroment which has different model of IGD
device.
Then I can pass through the device!!! again and again!!!
There's a couple of errors in dmesg:
[ 213.299260] DMAR: DRHD: handling fault status reg 2
[ 213.299266] DMAR: [DMA Write] Request device [00:02.0] fault addr 0
[fault reason 02] Present bit in context entry is clear
Is it a danger?
And another question, I can't modify screen resolution in windows, only
640*480, where is the limitation?
Acewind <acewind at gmail.com> 于2018年8月24日周五 下午2:21写道：
> BTW, in ubuntu 18.04, if run level is set to 3 (no GUI), no DMAR errors
> found in dmesg when virsh create a vm with igd passthrough.
> dmesg only output two lines:
> [ 62.601026] L1TF CPU bug present and SMT on, data leak possible. See
> CVE-2018-3646 and
> https://www.kernel.org/doc/html/latest/admin-guide/l1tf.html for details.
> [ 63.002568] pmd_set_huge: Cannot satisfy [mem 0xb7c00000-0xb7e00000]
> with a huge-page mapping due to MTRR override.
>
> I found an related option of qemu-system-x86_64:
> igd-passthru=on|off controls IGD GFX passthrough support (default=off)
> but nothing changes when set it to on.
>
> Alex Williamson <alex.williamson at redhat.com> 于2018年8月24日周五 上午3:32写道：
>
>> On Thu, 23 Aug 2018 15:57:11 +0800
>> acewind <acewind at gmail.com> wrote:
>>
>> > Yesterday I tried IGD with Ubuntu 18.04 but failed:
>> > https://www.redhat.com/archives/vfio-users/2018-August/msg00026.html
>> >
>> > Today I use the same hardware enviroment, reinstall OS to CentOS 7.3
>> > without GUI, upgrade kernel and qemu version:
>> >
>> > CPU: i3-4010U, support vt-d.
>> >
>> > [root at test ~]# lspci | grep VGA
>> > 00:02.0 VGA compatible controller: Intel Corporation Haswell-ULT
>> Integrated
>> > Graphics Controller (rev 09)
>> > (Only one IGD with vga and hdmi interface)
>> >
>> > [root at test ~]# uname -r
>> > 4.18.3-1.el7.centos.x86_64
>> >
>> > [root at test ~]# /usr/libexec/qemu-kvm --version
>> > QEMU emulator version 2.10.0(qemu-kvm-ev-2.10.0-21.el7_5.4.1)
>> > Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers
>>
>> You might be going further off track with this, the official RH
>> products do not support IGD assignment in any way other than kvmgt, ie.
>> GVT-g, ie. vGPU. In fact the RHEL kernel specifically disables some of
>> the IGD and VGA features necessary for legacy mode IGD assignment. I
>> don't know how much of this Centos carries to their product, but if
>> you're moving to Centos with the expectation that IGD assignment is
>> better supported there from the RH lineage, it's not.
>>
>> > Then I blacklist i915, forbid *fb:
>> >
>> > [root at test ~]# cat /etc/default/grub | grep GRUB_CMDLINE_LINUX
>> > GRUB_CMDLINE_LINUX="crashkernel=auto rhgb quiet intel_iommu=on iommu=pt
>> > video=vesafb:off vga=normal nofb nomodeset i915.modeset=0
>> > nouveau.modeset=0 rd.driver.blacklist=nouveau,i915 plymouth.ignore-udev"
>> >
>> > [root at test ~]# cat /etc/modprobe.d/blacklist-i915.conf
>> > install i915 /bin/true
>> >
>> > After rebooted:
>> > [root at test ~]# cat /proc/cmdline
>> > BOOT_IMAGE=/vmlinuz-4.18.3-1.el7.centos.x86_64
>> > root=UUID=78feccee-2f23-4ab6-b386-2b51d1664da7 ro crashkernel=auto rhgb
>> > quiet intel_iommu=on iommu=pt video=vesafb:off vga=normal nofb nomodeset
>> > i915.modeset=0 nouveau.modeset=0 rd.driver.blacklist=nouveau,i915
>> > plymouth.ignore-udev
>> >
>> > [root at test ~]# cat /proc/iomem | grep fb
>> > 01c031d1-02463fbf : Kernel data
>> > f8000000-fbffffff : PCI MMCONFIG 0000 [bus 00-3f]
>> > f8000000-fbffffff : Reserved
>> > f8000000-fbffffff : pnp 00:05
>> >
>> > [root at test ~]# lsmod | grep i915
>> > ... nothing ...
>>
>> It's really not necessary to go to these sorts of lengths to disable
>> i915. You can simply 'virsh nodedev-detach pci_0000_00_02_0'. IME,
>> the place where i915 causes trouble is if you later attempt to re-bind
>> the device to i915.
>>
>> > Now I begin to load vfio modules.
>> >
>> > [root at test ~]# modprobe vfio
>> > [root at test ~]# modprobe vfio-pci
>> >
>> > [root at test ~]# echo "vfio-pci" >
>> > /sys/bus/pci/devices/0000:00:02.0/driver_override
>> > [root at test ~]# ./vfio-pci-bind.sh 0000:00:02.0
>> > (script ref:
>> >
>> https://github.com/andre-richter/vfio-pci-bind/blob/master/vfio-pci-bind.sh
>> )
>> >
>> > [root at test ~]# lspci -v -s 00:02.0
>> > 00:02.0 VGA compatible controller: Intel Corporation Haswell-ULT
>> Integrated
>> > Graphics Controller (rev 09) (prog-if 00 [VGA controller])
>> > Subsystem: Intel Corporation Haswell-ULT Integrated Graphics
>> Controller
>> > Flags: fast devsel, IRQ 16
>> > Memory at a7c00000 (64-bit, non-prefetchable) [size=4M]
>> > Memory at b0000000 (64-bit, prefetchable) [size=256M]
>> > I/O ports at 4000 [size=64]
>> > [virtual] Expansion ROM at 000c0000 [disabled] [size=128K]
>> > Capabilities: [90] MSI: Enable- Count=1/1 Maskable- 64bit-
>> > Capabilities: [d0] Power Management version 2
>> > Capabilities: [a4] PCI Advanced Features
>> > Kernel driver in use: vfio-pci
>> > Kernel modules: i915
>> >
>> > Now virsh create a win7 vm, xml file is:
>> >
>> > <domain type='kvm' id='3' xmlns:qemu='
>> > http://libvirt.org/schemas/domain/qemu/1.0'>
>> > <name>windows_7_ultimate_with_sp1_x64</name>
>> > <uuid>2fcccb90-f622-481a-8819-367c642c0a09</uuid>
>> > <memory unit='KiB'>2097152</memory>
>> > <currentMemory unit='KiB'>2097152</currentMemory>
>> > <vcpu placement='static'>2</vcpu>
>> > <resource>
>> > <partition>/machine</partition>
>> > </resource>
>> > <os>
>> > <type arch='x86_64' machine='pc-i440fx-rhel7.5.0'>hvm</type>
>> > <boot dev='hd'/>
>> > </os>
>> > <features>
>> > <acpi/>
>> > <apic/>
>> > <hyperv>
>> > <relaxed state='on'/>
>> > <vapic state='on'/>
>> > <spinlocks state='on' retries='8191'/>
>> > </hyperv>
>> > </features>
>> > <cpu mode='custom' match='exact' check='full'>
>> > <model fallback='forbid'>Haswell-noTSX</model>
>> > <feature policy='require' name='vme'/>
>> > <feature policy='require' name='f16c'/>
>> > <feature policy='require' name='rdrand'/>
>> > <feature policy='require' name='hypervisor'/>
>> > <feature policy='require' name='arat'/>
>> > <feature policy='require' name='xsaveopt'/>
>> > <feature policy='require' name='abm'/>
>> > </cpu>
>> > <clock offset='localtime'>
>> > <timer name='rtc' tickpolicy='catchup'/>
>> > <timer name='pit' tickpolicy='delay'/>
>> > <timer name='hpet' present='no'/>
>> > <timer name='hypervclock' present='yes'/>
>> > </clock>
>> > <on_poweroff>destroy</on_poweroff>
>> > <on_reboot>restart</on_reboot>
>> > <on_crash>destroy</on_crash>
>> > <pm>
>> > <suspend-to-mem enabled='no'/>
>> > <suspend-to-disk enabled='no'/>
>> > </pm>
>> > <devices>
>> > <emulator>/usr/libexec/qemu-kvm</emulator>
>> > <disk type='file' device='disk'>
>> > <driver name='qemu' type='qcow2'/>
>> > <source file='/home/windows_7_ultimate_with_sp1_x64.qcow2'/>
>> > <backingStore/>
>> > <target dev='hda' bus='ide'/>
>> > <alias name='ide0-0-0'/>
>> > <address type='drive' controller='0' bus='0' target='0' unit='0'/>
>> > </disk>
>> >
>> > <hostdev mode='subsystem' type='pci' managed='yes'>
>> > <driver name='vfio'/>
>> > <source>
>> > <address domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
>> > </source>
>> > <alias name='hostdev0'/>
>> > <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
>> > function='0x0'/>
>> > </hostdev>
>> > </devices>
>> >
>> > <qemu:commandline>
>> > <qemu:arg value='-bios'/>
>> > <qemu:arg value='/usr/share/seabios/bios.bin'/>
>> > <qemu:arg value='-chardev'/>
>> > <qemu:arg value='file,id=seabios,path=/tmp/bios.log'/>
>> > <qemu:arg value='-device'/>
>> > <qemu:arg value='isa-debugcon,iobase=0x402,chardev=seabios'/>
>> > </qemu:commandline>
>> > </domain>
>> >
>> > After created, the vm is running, monitor screen turn to black then no
>> > output.
>> > The end of seabios log is:
>> >
>> > ...
>> > Scan for VGA option rom
>> > Running option rom at c000:0003
>> >
>> > Any problem with VGA rom? In the BIOS I have already enabled CSM and the
>> > option rom of video is set to legacy.
>> >
>> > Ref to
>> http://lists.gnu.org/archive/html/qemu-discuss/2018-04/msg00052.html
>> > and https://github.com/awilliam/rom-parser
>> >
>> > I dump the vga rom:
>> >
>> > [root at test rom]# echo 1 > /sys/devices/pci0000:00/0000:00:02.0/rom
>> > [root at test rom]# cat /sys/devices/pci0000:00/0000:00:02.0/rom >
>> vbios.dump
>> > [root at test rom]# echo 0 > /sys/devices/pci0000:00/0000:00:02.0/rom
>> >
>> > [root at test rom]# ./rom-parser/rom-parser vbios.dump
>> > Valid ROM signature found @0h, PCIR offset 40h
>> > PCIR: type 0 (x86 PC-AT), vendor: 8086, device: 0406, class: 030000
>> > PCIR: revision 3, vendor revision: 0
>> > Last image
>> >
>> > [root at test rom]# lspci -nns 00:02.0
>> > 00:02.0 VGA compatible controller [0300]: Intel Corporation Haswell-ULT
>> > Integrated Graphics Controller [8086:0a16] (rev 09)
>> >
>> > The device code in the rom is diffrent from vga device? So I modify it
>> with
>> > rom-fixer command:
>> >
>> > [root at test rom]# ./rom-parser/rom-fixer ./vbios.dump
>> > Valid ROM signature found @0h, PCIR offset 40h
>> > PCIR: type 0 (x86 PC-AT), vendor: 8086, device: 0406, class: 030000
>> > PCIR: revision 3, vendor revision: 0
>> >
>> > Modify vendor ID 8086? (y/n): y
>> > New vendor ID: 8086
>> > Overwrite vendor ID with 8086? (y/n): y
>> > Modify device ID 0406? (y/n): y
>> > New device ID: 0a16
>> > Overwrite device ID with 0a16? (y/n): y
>> > Last image
>> > ROM checksum is invalid, fix? (y/n): y
>>
>> The only reason this should be necessary is if you're already using an
>> externally provided rom via <rom>. QEMU will do the above for you
>> automatically when it's reading the rom directly from the device. So
>> there's no net effect of this change.
>>
>> > Then I edit the vm xml, and romfile args to hostdev:
>> > <rom file='/path/to/vbios.dump'/>
>> >
>> > Then destroy and create the vm again, still black screen ...
>> >
>> > Is there any more methods to debug？
>>
>> I'd use upstream kernel and QEMU. Your previous post indicated DMAR
>> faults. A couple of those are normal, but you seem to have a fair
>> number of them and the address is incrementing as if our attempt to
>> trap and relocate stolen memory isn't working. It's possible this
>> particular GPU uses a different page table format that we don't know
>> how to handle yet. Thanks,
>>
>> Alex
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20180824/a19f3b81/attachment.htm>
More information about the vfio-users
mailing list