<div dir="ltr">Today I return to use Ubuntu 18.04.1 LTS:<div><div>root@test:~# lsb_release -a</div><div>No LSB modules are available.</div><div>Distributor ID:<span style="white-space:pre"> </span>Ubuntu</div><div>Description:<span style="white-space:pre"> </span>Ubuntu 18.04.1 LTS</div><div>Release:<span style="white-space:pre"> </span>18.04</div><div>Codename:<span style="white-space:pre"> </span>bionic</div></div><div><br></div><div><div>root@test:~# uname -a</div><div>Linux idv 4.15.0-33-generic #36-Ubuntu SMP Wed Aug 15 16:00:05 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux</div><div><br></div><div>root@test:~# uname -r</div><div>4.15.0-33-generic</div><div><br></div><div>root@test:~# qemu-system-x86_64 --version</div><div>QEMU emulator version 2.11.1(Debian 1:2.11+dfsg-1ubuntu7.4)</div><div>Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers</div><div><br></div></div><div>Prepare as what I did before ... then virsh create, black screen again, dmesg shows some exception in kernel:</div><div><br></div><div><div>[ 482.998862] L1TF CPU bug present and SMT on, data leak possible. See CVE-2018-3646 and <a href="https://www.kernel.org/doc/html/latest/admin-guide/l1tf.html">https://www.kernel.org/doc/html/latest/admin-guide/l1tf.html</a> for details.</div><div>[ 483.495986] pmd_set_huge: Cannot satisfy [mem 0xb7c00000-0xb7e00000] with a huge-page mapping due to MTRR override.</div><div>[ 484.310469] general protection fault: 0000 [#1] SMP PTI</div><div>[ 484.310502] Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack libcrc32c ipt_REJECT nf_reject_ipv4 xt_tcpudp bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd glue_helper cryptd intel_cstate intel_rapl_perf snd_hda_codec_realtek snd_hda_codec_generic snd_seq_midi snd_seq_midi_event snd_hda_intel snd_hda_codec snd_rawmidi joydev snd_hda_core snd_hwdep snd_pcm input_leds snd_seq lpc_ich shpchp snd_seq_device snd_timer snd binfmt_misc soundcore kvm_intel kvm acpi_pad mac_hid sch_fq_codel vfio_pci</div><div>[ 484.310661] vfio_virqfd irqbypass vfio_iommu_type1 vfio parport_pc ppdev lp parport ip_tables x_tables autofs4 hid_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect r8169 sysimgblt fb_sys_fops mii usbhid ahci drm libahci hid video</div><div>[ 484.310732] CPU: 0 PID: 1272 Comm: CPU 0/KVM Not tainted 4.15.0-33-generic #36-Ubuntu</div><div>[ 484.310764] Hardware name: To be filled by O.E.M. To be filled by O.E.M./CRESCENTBAY, BIOS 5.011 09/02/2016</div><div>[ 484.310801] RIP: 0010:is_invalid_reserved_pfn+0x6e/0xa0 [vfio_iommu_type1]</div><div>[ 484.310832] RSP: 0018:ffffa692c11d7c68 EFLAGS: 00010202</div><div>[ 484.310871] RAX: 5710738949085b89 RBX: ffff8950abb739c0 RCX: ffff8950b4fff000</div><div>[ 484.310910] RDX: 5710738949085b88 RSI: 0000000000000000 RDI: ffffd06582df0000</div><div>[ 484.310950] RBP: ffffa692c11d7cb0 R08: 0000000000000007 R09: ffffa692c11d7c70</div><div>[ 484.310989] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000000</div><div>[ 484.311029] R13: ffffa692c11d7da8 R14: 00000000fffffff2 R15: ffff8950abb73a30</div><div>[ 484.311069] FS: 00007f7b23efe700(0000) GS:ffff8950b2e00000(0000) knlGS:0000000000000000</div><div>[ 484.311119] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033</div><div>[ 484.311157] CR2: 00007f7b1c0cf000 CR3: 00000000ac350004 CR4: 00000000001626f0</div><div>[ 484.311196] Call Trace:</div><div>[ 484.311234] ? vaddr_get_pfn+0x185/0x200 [vfio_iommu_type1]</div><div>[ 484.311275] vfio_pin_pages_remote+0x71/0x3b0 [vfio_iommu_type1]</div><div>[ 484.311315] vfio_iommu_type1_ioctl+0x395/0x905 [vfio_iommu_type1]</div><div>[ 484.311358] vfio_fops_unl_ioctl+0x6b/0x270 [vfio]</div><div>[ 484.311398] do_vfs_ioctl+0xa8/0x630</div><div>[ 484.311436] ? handle_mm_fault+0xb1/0x1f0</div><div>[ 484.311475] ? __do_page_fault+0x270/0x4d0</div><div>[ 484.311512] SyS_ioctl+0x79/0x90</div><div>[ 484.311550] do_syscall_64+0x73/0x130</div><div>[ 484.311588] entry_SYSCALL_64_after_hwframe+0x3d/0xa2</div><div>[ 484.311626] RIP: 0033:0x7f7b322af5d7</div><div>[ 484.311661] RSP: 002b:00007f7b23efd298 EFLAGS: 00000246 ORIG_RAX: 0000000000000010</div><div>[ 484.311711] RAX: ffffffffffffffda RBX: 0000558737d5ae80 RCX: 00007f7b322af5d7</div><div>[ 484.311751] RDX: 00007f7b23efd2a0 RSI: 0000000000003b71 RDI: 0000000000000012</div><div>[ 484.311790] RBP: 00000000fe400000 R08: 0000000000000000 R09: 0000000000000000</div><div>[ 484.311830] R10: 00000000fe400000 R11: 0000000000000246 R12: 0000000000400000</div><div>[ 484.311869] R13: 00007f7b23efd2a0 R14: 0000000000000000 R15: 00000000fe7fffff</div><div>[ 484.311909] Code: 02 02 75 01 c3 48 c1 e7 06 48 03 3d 35 ae 6a f4 48 8b 47 20 a8 01 75 0f 48 8b 07 48 8b 07 48 c1 e8 0b 83 e0 01 eb dd 48 8d 50 ff <48> 8b 40 ff 48 c1 e8 0b 83 e0 01 48 39 d7 74 e0 55 48 89 e5 48</div><div>[ 484.312024] RIP: is_invalid_reserved_pfn+0x6e/0xa0 [vfio_iommu_type1] RSP: ffffa692c11d7c68</div><div>[ 484.312090] ---[ end trace 00f76f6e80792f13 ]---</div></div><div><br></div><div>The vm is running and there's no other error in libvirt qemu log:</div><div><br></div><div><div>root@test:~# ps -elf | grep qemu</div><div>6 S libvirt+ 1257 1 0 80 0 - 703903 futex_ 09:46 ? 00:00:01 qemu-system-x86_64 -enable-kvm -name guest=windows_7_ultimate_with_sp1_x64,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-windows_7_ultimate_w/master-key.aes -machine pc-i440fx-bionic,accel=kvm,usb=off,dump-guest-core=off -cpu Haswell-noTSX-IBRS,vme=on,f16c=on,rdrand=on,hypervisor=on,arat=on,xsaveopt=on,abm=on,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff -m 2048 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid 963d8c33-0b99-45a8-91de-16c22975a51a -display none -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-1-windows_7_ultimate_w/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x4.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x4 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x4.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x4.0x2 -drive file=/home/windows_7_ultimate_with_sp1_x64.qcow2,format=qcow2,if=none,id=drive-ide0-0-0 -device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -netdev tap,fd=26,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:b8:20:01,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0,bus=usb.0,port=1 <u>-device vfio-pci,host=00:02.0,id=hostdev0,bus=pci.0,addr=0x2</u> -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on</div></div><div><br></div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr">Alex Williamson <<a href="mailto:alex.williamson@redhat.com">alex.williamson@redhat.com</a>> 于2018年8月24日周五 上午3:32写道：<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On Thu, 23 Aug 2018 15:57:11 +0800<br>
acewind <<a href="mailto:acewind@gmail.com" target="_blank">acewind@gmail.com</a>> wrote:<br>
<br>
> Yesterday I tried IGD with Ubuntu 18.04 but failed:<br>
> <a href="https://www.redhat.com/archives/vfio-users/2018-August/msg00026.html" rel="noreferrer" target="_blank">https://www.redhat.com/archives/vfio-users/2018-August/msg00026.html</a><br>
> <br>
> Today I use the same hardware enviroment, reinstall OS to CentOS 7.3<br>
> without GUI, upgrade kernel and qemu version:<br>
> <br>
> CPU: i3-4010U, support vt-d.<br>
> <br>
> [root@test ~]# lspci | grep VGA<br>
> 00:02.0 VGA compatible controller: Intel Corporation Haswell-ULT Integrated<br>
> Graphics Controller (rev 09)<br>
> (Only one IGD with vga and hdmi interface)<br>
> <br>
> [root@test ~]# uname -r<br>
> 4.18.3-1.el7.centos.x86_64<br>
> <br>
> [root@test ~]# /usr/libexec/qemu-kvm --version<br>
> QEMU emulator version 2.10.0(qemu-kvm-ev-2.10.0-21.el7_5.4.1)<br>
> Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers<br>
<br>
You might be going further off track with this, the official RH<br>
products do not support IGD assignment in any way other than kvmgt, ie.<br>
GVT-g, ie. vGPU. In fact the RHEL kernel specifically disables some of<br>
the IGD and VGA features necessary for legacy mode IGD assignment. I<br>
don't know how much of this Centos carries to their product, but if<br>
you're moving to Centos with the expectation that IGD assignment is<br>
better supported there from the RH lineage, it's not.<br>
<br>
> Then I blacklist i915, forbid *fb:<br>
> <br>
> [root@test ~]# cat /etc/default/grub | grep GRUB_CMDLINE_LINUX<br>
> GRUB_CMDLINE_LINUX="crashkernel=auto rhgb quiet intel_iommu=on iommu=pt<br>
> video=vesafb:off vga=normal nofb nomodeset i915.modeset=0<br>
> nouveau.modeset=0 rd.driver.blacklist=nouveau,i915 plymouth.ignore-udev"<br>
> <br>
> [root@test ~]# cat /etc/modprobe.d/blacklist-i915.conf<br>
> install i915 /bin/true<br>
> <br>
> After rebooted:<br>
> [root@test ~]# cat /proc/cmdline<br>
> BOOT_IMAGE=/vmlinuz-4.18.3-1.el7.centos.x86_64<br>
> root=UUID=78feccee-2f23-4ab6-b386-2b51d1664da7 ro crashkernel=auto rhgb<br>
> quiet intel_iommu=on iommu=pt video=vesafb:off vga=normal nofb nomodeset<br>
> i915.modeset=0 nouveau.modeset=0 rd.driver.blacklist=nouveau,i915<br>
> plymouth.ignore-udev<br>
> <br>
> [root@test ~]# cat /proc/iomem | grep fb<br>
> 01c031d1-02463fbf : Kernel data<br>
> f8000000-fbffffff : PCI MMCONFIG 0000 [bus 00-3f]<br>
> f8000000-fbffffff : Reserved<br>
> f8000000-fbffffff : pnp 00:05<br>
> <br>
> [root@test ~]# lsmod | grep i915<br>
> ... nothing ...<br>
<br>
It's really not necessary to go to these sorts of lengths to disable<br>
i915. You can simply 'virsh nodedev-detach pci_0000_00_02_0'. IME,<br>
the place where i915 causes trouble is if you later attempt to re-bind<br>
the device to i915.<br>
<br>
> Now I begin to load vfio modules.<br>
> <br>
> [root@test ~]# modprobe vfio<br>
> [root@test ~]# modprobe vfio-pci<br>
> <br>
> [root@test ~]# echo "vfio-pci" ><br>
> /sys/bus/pci/devices/0000:00:02.0/driver_override<br>
> [root@test ~]# ./vfio-pci-bind.sh 0000:00:02.0<br>
> (script ref:<br>
> <a href="https://github.com/andre-richter/vfio-pci-bind/blob/master/vfio-pci-bind.sh" rel="noreferrer" target="_blank">https://github.com/andre-richter/vfio-pci-bind/blob/master/vfio-pci-bind.sh</a>)<br>
> <br>
> [root@test ~]# lspci -v -s 00:02.0<br>
> 00:02.0 VGA compatible controller: Intel Corporation Haswell-ULT Integrated<br>
> Graphics Controller (rev 09) (prog-if 00 [VGA controller])<br>
> Subsystem: Intel Corporation Haswell-ULT Integrated Graphics Controller<br>
> Flags: fast devsel, IRQ 16<br>
> Memory at a7c00000 (64-bit, non-prefetchable) [size=4M]<br>
> Memory at b0000000 (64-bit, prefetchable) [size=256M]<br>
> I/O ports at 4000 [size=64]<br>
> [virtual] Expansion ROM at 000c0000 [disabled] [size=128K]<br>
> Capabilities: [90] MSI: Enable- Count=1/1 Maskable- 64bit-<br>
> Capabilities: [d0] Power Management version 2<br>
> Capabilities: [a4] PCI Advanced Features<br>
> Kernel driver in use: vfio-pci<br>
> Kernel modules: i915<br>
> <br>
> Now virsh create a win7 vm, xml file is:<br>
> <br>
> <domain type='kvm' id='3' xmlns:qemu='<br>
> <a href="http://libvirt.org/schemas/domain/qemu/1.0" rel="noreferrer" target="_blank">http://libvirt.org/schemas/domain/qemu/1.0</a>'> <br>
> <name>windows_7_ultimate_with_sp1_x64</name><br>
> <uuid>2fcccb90-f622-481a-8819-367c642c0a09</uuid><br>
> <memory unit='KiB'>2097152</memory><br>
> <currentMemory unit='KiB'>2097152</currentMemory><br>
> <vcpu placement='static'>2</vcpu><br>
> <resource><br>
> <partition>/machine</partition><br>
> </resource><br>
> <os><br>
> <type arch='x86_64' machine='pc-i440fx-rhel7.5.0'>hvm</type><br>
> <boot dev='hd'/><br>
> </os><br>
> <features><br>
> <acpi/><br>
> <apic/><br>
> <hyperv><br>
> <relaxed state='on'/><br>
> <vapic state='on'/><br>
> <spinlocks state='on' retries='8191'/><br>
> </hyperv><br>
> </features><br>
> <cpu mode='custom' match='exact' check='full'><br>
> <model fallback='forbid'>Haswell-noTSX</model><br>
> <feature policy='require' name='vme'/><br>
> <feature policy='require' name='f16c'/><br>
> <feature policy='require' name='rdrand'/><br>
> <feature policy='require' name='hypervisor'/><br>
> <feature policy='require' name='arat'/><br>
> <feature policy='require' name='xsaveopt'/><br>
> <feature policy='require' name='abm'/><br>
> </cpu><br>
> <clock offset='localtime'><br>
> <timer name='rtc' tickpolicy='catchup'/><br>
> <timer name='pit' tickpolicy='delay'/><br>
> <timer name='hpet' present='no'/><br>
> <timer name='hypervclock' present='yes'/><br>
> </clock><br>
> <on_poweroff>destroy</on_poweroff><br>
> <on_reboot>restart</on_reboot><br>
> <on_crash>destroy</on_crash><br>
> <pm><br>
> <suspend-to-mem enabled='no'/><br>
> <suspend-to-disk enabled='no'/><br>
> </pm><br>
> <devices><br>
> <emulator>/usr/libexec/qemu-kvm</emulator><br>
> <disk type='file' device='disk'><br>
> <driver name='qemu' type='qcow2'/><br>
> <source file='/home/windows_7_ultimate_with_sp1_x64.qcow2'/><br>
> <backingStore/><br>
> <target dev='hda' bus='ide'/><br>
> <alias name='ide0-0-0'/><br>
> <address type='drive' controller='0' bus='0' target='0' unit='0'/><br>
> </disk><br>
> <br>
> <hostdev mode='subsystem' type='pci' managed='yes'><br>
> <driver name='vfio'/><br>
> <source><br>
> <address domain='0x0000' bus='0x00' slot='0x02' function='0x0'/><br>
> </source><br>
> <alias name='hostdev0'/><br>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x02'<br>
> function='0x0'/> <br>
> </hostdev><br>
> </devices><br>
> <br>
> <qemu:commandline><br>
> <qemu:arg value='-bios'/><br>
> <qemu:arg value='/usr/share/seabios/bios.bin'/><br>
> <qemu:arg value='-chardev'/><br>
> <qemu:arg value='file,id=seabios,path=/tmp/bios.log'/><br>
> <qemu:arg value='-device'/><br>
> <qemu:arg value='isa-debugcon,iobase=0x402,chardev=seabios'/><br>
> </qemu:commandline><br>
> </domain><br>
> <br>
> After created, the vm is running, monitor screen turn to black then no<br>
> output.<br>
> The end of seabios log is:<br>
> <br>
> ...<br>
> Scan for VGA option rom<br>
> Running option rom at c000:0003<br>
> <br>
> Any problem with VGA rom? In the BIOS I have already enabled CSM and the<br>
> option rom of video is set to legacy.<br>
> <br>
> Ref to <a href="http://lists.gnu.org/archive/html/qemu-discuss/2018-04/msg00052.html" rel="noreferrer" target="_blank">http://lists.gnu.org/archive/html/qemu-discuss/2018-04/msg00052.html</a><br>
> and <a href="https://github.com/awilliam/rom-parser" rel="noreferrer" target="_blank">https://github.com/awilliam/rom-parser</a><br>
> <br>
> I dump the vga rom:<br>
> <br>
> [root@test rom]# echo 1 > /sys/devices/pci0000:00/0000:00:02.0/rom<br>
> [root@test rom]# cat /sys/devices/pci0000:00/0000:00:02.0/rom > vbios.dump<br>
> [root@test rom]# echo 0 > /sys/devices/pci0000:00/0000:00:02.0/rom<br>
> <br>
> [root@test rom]# ./rom-parser/rom-parser vbios.dump<br>
> Valid ROM signature found @0h, PCIR offset 40h<br>
> PCIR: type 0 (x86 PC-AT), vendor: 8086, device: 0406, class: 030000<br>
> PCIR: revision 3, vendor revision: 0<br>
> Last image<br>
> <br>
> [root@test rom]# lspci -nns 00:02.0<br>
> 00:02.0 VGA compatible controller [0300]: Intel Corporation Haswell-ULT<br>
> Integrated Graphics Controller [8086:0a16] (rev 09)<br>
> <br>
> The device code in the rom is diffrent from vga device? So I modify it with<br>
> rom-fixer command:<br>
> <br>
> [root@test rom]# ./rom-parser/rom-fixer ./vbios.dump<br>
> Valid ROM signature found @0h, PCIR offset 40h<br>
> PCIR: type 0 (x86 PC-AT), vendor: 8086, device: 0406, class: 030000<br>
> PCIR: revision 3, vendor revision: 0<br>
> <br>
> Modify vendor ID 8086? (y/n): y<br>
> New vendor ID: 8086<br>
> Overwrite vendor ID with 8086? (y/n): y<br>
> Modify device ID 0406? (y/n): y<br>
> New device ID: 0a16<br>
> Overwrite device ID with 0a16? (y/n): y<br>
> Last image<br>
> ROM checksum is invalid, fix? (y/n): y<br>
<br>
The only reason this should be necessary is if you're already using an<br>
externally provided rom via <rom>. QEMU will do the above for you<br>
automatically when it's reading the rom directly from the device. So<br>
there's no net effect of this change.<br>
<br>
> Then I edit the vm xml, and romfile args to hostdev:<br>
> <rom file='/path/to/vbios.dump'/><br>
> <br>
> Then destroy and create the vm again, still black screen ...<br>
> <br>
> Is there any more methods to debug？<br>
<br>
I'd use upstream kernel and QEMU. Your previous post indicated DMAR<br>
faults. A couple of those are normal, but you seem to have a fair<br>
number of them and the address is incrementing as if our attempt to<br>
trap and relocate stolen memory isn't working. It's possible this<br>
particular GPU uses a different page table format that we don't know<br>
how to handle yet. Thanks,<br>
<br>
Alex<br>
</blockquote></div>