<html><head><meta http-equiv="content-type" content="text/html; charset=ISO-8859-1"><style>body { line-height: 1.5; }blockquote { margin-top: 0px; margin-bottom: 0px; margin-left: 0.5em; }body { font-size: 12pt; font-family: ????; color: rgb(0, 0, 0); line-height: 1.5; }</style></head><body>
<div><span></span>thanks,Alex.</div><div><br></div><div>I use the same config to launch the VMs in the same time.Card with PCI address 02:00.0 and the 03:00.0 works fine, 82:00.0 and 83:00.0 are not.</div><div><br></div><div>I just check the BIOS settings.Configuration of PCI-e device is the same.</div><div><br></div><div>here is the lspci command output:</div><div><br></div><div><span style="background-color: rgba(0, 0, 0, 0); font-size: 12pt; line-height: 1.5;"> </span><span style="background-color: rgba(0, 0, 0, 0); font-size: 12pt; line-height: 1.5;"> </span><span style="background-color: rgba(0, 0, 0, 0); font-size: 12pt; line-height: 1.5;">02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)</span></div><div><span style="color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);"> Subsystem: LeadTek Research Inc. Device [107d:2762]<br> Kernel driver in use: vfio-pci<br> Kernel modules: nouveau</span></div><div><span style="color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);"> </span><span style="background-color: rgba(0, 0, 0, 0); font-size: 12pt; line-height: 1.5;"> .......</span></div><div><span style="color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);"> </span><span style="background-color: rgba(0, 0, 0, 0); font-size: 12pt; line-height: 1.5;"> </span><span style="background-color: rgba(0, 0, 0, 0); font-size: 12pt; line-height: 1.5;">03:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)</span></div><span style="color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);"> Subsystem: LeadTek Research Inc. Device [107d:2762]<br> Kernel driver in use: vfio-pci<br> Kernel modules: nouveau</span><div><div><span style="color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);"> </span><span style="background-color: rgba(0, 0, 0, 0); font-size: 12pt; line-height: 1.5;"> ......</span></div><div><span style="color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);"> </span><span style="background-color: rgba(0, 0, 0, 0); font-size: 12pt; line-height: 1.5;"> </span><span style="background-color: rgba(0, 0, 0, 0); font-size: 12pt; line-height: 1.5;">82:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)</span></div><span style="color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);"> Subsystem: LeadTek Research Inc. Device [107d:2762]<br> Kernel driver in use: vfio-pci<br> Kernel modules: nouveau</span></div><div><span style="color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);"> </span><span style="color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);"> ......</span></div><div><span style="color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);"> </span><span style="color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);"> </span><span style="color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);">83:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)<br> Subsystem: LeadTek Research Inc. Device [107d:2762]<br> Kernel driver in use: vfio-pci<br> Kernel modules: nouveau</span><br><hr style="width: 210px; height: 1px; display: none;" color="#b5c4df" size="1" align="left">
<div><span></span></div>
<blockquote style="margin-top: 0px; margin-bottom: 0px; margin-left: 0.5em;"><div> </div><div>I also edit the /usr/bin/qemu-kvm like this:</div><div><span style="color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);"> </span></div><span style="color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);"> </span><span style="color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);">#!/bin/sh<br></span><span style="color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);"> </span><span style="color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);">result="$@"<br></span><span style="color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);"> </span><span style="color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);">result=${result/02:00.0/02:00.0,x-vga=on}<br></span><span style="color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);"> </span><span style="color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);">result=${result/03:00.0/03:00.0,x-vga=on}<br></span><span style="color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);"> </span><span style="color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);">result=${result/82:00.0/82:00.0,x-vga=on}<br></span><span style="color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);"> </span><span style="color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);">result=${result/83:00.0/83:00.0,x-vga=on}<br></span><span style="color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);"><br></span><span style="color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);"> </span><span style="color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);">exec /usr/bin/qemu-system-x86_64 -machine accel=kvm $result</span><div><br></div><div>in F22,the original /usr/bin/qemu-kvm is a script:</div><div><span style="background-color: rgba(0, 0, 0, 0); font-size: 12pt; line-height: 1.5;"> </span></div><span style="color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);"> </span><span style="color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);">#!/bin/sh<br></span><span style="color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);"> </span><span style="color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);"><br></span><span style="color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);"> </span><span style="color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);">exec /usr/bin/qemu-system-x86_64 -machine accel=kvm </span>$@<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm"><div style="PADDING-RIGHT: 8px; PADDING-LEFT: 8px; FONT-SIZE: 12px;FONT-FAMILY:tahoma;COLOR:#000000; BACKGROUND: #efefef; PADDING-BOTTOM: 8px; PADDING-TOP: 8px"><div><b>From:</b> <a href="mailto:alex.williamson@redhat.com">Alex Williamson</a></div><div><b>Date:</b> 2016-05-06 23:12</div><div><b>To:</b> <a href="mailto:Kayv@qq.com">xuemin.wen</a></div><div><b>CC:</b> <a href="mailto:vfio-users@redhat.com">vfio-users</a></div><div><b>Subject:</b> Re: [vfio-users] KVM internal error. Suberror: 1, emulation failure problem</div></div></div><div><div>On Fri, 6 May 2016 22:56:14 +0800</div>
<div>"xuemin.wen" <Kayv@qq.com> wrote:</div>
<div> </div>
<div>> hello everyone,</div>
<div>> I am trying to passthrough 4*GTX970 video card to individual VMs,here is the problem.</div>
<div>> </div>
<div>> the GTX970 with PCI address 02:00.0 and 03:00.0 can passthrough to the VMs,and works good.</div>
<div>> </div>
<div>> the GTX970 with PCI address 82:00.0 and 83:00.0 also can passthrough to the VMs,but the VMs just work for a few seconds with CPU usage 100%,and return a error:"KVM internal error. Suberror: 1 emulation failure"</div>
<div>> </div>
<div>> if i don't passtrough GTX 970,there is no error. So I think the issue with the relevant VFIO,not the QEMU-KVM. </div>
<div>> </div>
<div>> my host hardware is :</div>
<div>> Supermicro 1028GQ-TR system include X10DGQ motherboard</div>
<div>> XEON E5 2630v3 * 2</div>
<div>> 16GB RAM * 4</div>
<div>> LEADTEK GTX970 * 4</div>
<div>> INTEL SSD S3510</div>
<div>> </div>
<div>> my host system is :</div>
<div>> Fedora 22 server@4.4.5-200.fc22.x86_64</div>
<div>> QEMU emulator version 2.3.1 (qemu-2.3.1-12.fc22)</div>
<div>> libvirt version: 1.2.13.2</div>
<div>> </div>
<div>> the log of GTX with PCI address 82:00.0 or 83:00.0 </div>
<div>> </div>
<div>> 2016-05-06 13:37:10.197+0000: starting up libvirt version: 1.2.13.2, package: 2.fc22 (Fedora Project, 2016-01-21-01:06:26, buildvm-06.phx2.fedoraproject.org), qemu version: 2.3.1 (qemu-2.3.1-12.fc22)</div>
<div>> LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/bin/qemu-kvm -name instance-000002ef -S -machine pc-i440fx-2.3,accel=kvm,usb=off -cpu Haswell-noTSX,+abm,+pdpe1gb,+rdrand,+f16c,+osxsave,+dca,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme,kvm=off -m 4096 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid 92af0144-107e-48e1-bed1-ad04fac1faea -smbios type=1,manufacturer=Fedora Project,product=OpenStack Nova,version=2014.2.4-1.fc22,serial=26348ff0-e2d1-459f-a026-ec33b800e05c,uuid=92af0144-107e-48e1-bed1-ad04fac1faea -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-000002ef.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/nova/insta!</div>
<div> nces/92af0144-107e-48e1-bed1-ad04fac1faea/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=27,id=hostnet0,vhost=on,vhostfd=29 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:30:0d:09,bus=pci.0,addr=0x2 -chardev file,id=charserial0,path=/var/lib/nova/instances/92af0144-107e-48e1-bed1-ad04fac1faea/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0 -device vfio-pci,host=83:00.0,id=hostdev0,bus=pci.0,addr=0x4 -device vfio-pci,host=83:00.1,id=hostdev1,bus=pci.0,addr=0x5 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -msg timestamp=on</div>
<div>> char device redirected to /dev/pts/3 (label charserial1)</div>
<div>> KVM internal error. Suberror: 1</div>
<div>> emulation failure</div>
<div>> EAX=00000000 EBX=00000000 ECX=00000000 EDX=00000000</div>
<div>> ESI=00000000 EDI=00000000 EBP=00000000 ESP=00006de8</div>
<div>> EIP=0006fdff EFL=00010006 [-----P-] CPL=0 II=0 A20=1 SMM=0 HLT=0</div>
<div>> ES =0000 00000000 ffffffff 00809300</div>
<div>> CS =3020 00030200 ffffffff 00809b00</div>
<div>> SS =0000 00000000 ffffffff 00809300</div>
<div>> DS =0000 00000000 ffffffff 00809300</div>
<div>> FS =0000 00000000 ffffffff 00809300</div>
<div>> GS =0000 00000000 ffffffff 00809300</div>
<div>> LDT=0000 00000000 0000ffff 00008200</div>
<div>> TR =0000 00000000 0000ffff 00008b00</div>
<div>> GDT= 000f6b70 00000037</div>
<div>> IDT= 00000000 000003ff</div>
<div>> CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000</div>
<div>> DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000</div>
<div>> DR6=00000000ffff0ff0 DR7=0000000000000400</div>
<div>> EFER=0000000000000000</div>
<div>> Code=00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff</div>
<div>> qemu: terminating on signal 15 from pid 1500</div>
<div>> 2016-05-06 13:44:57.035+0000: shutting down</div>
<div> </div>
<div>Seems like you're not using OVMF and you're not enabling VGA, so your</div>
<div>config is in some weird state where the VGA option ROM gets executed</div>
<div>but VGA resources don't actually get routed to the device. Not</div>
<div>terribly predictable what might happen in that case. If you have other</div>
<div>VMs working with this config, you might try dumping the ROM from those</div>
<div>cards and passing it as a file to these cards. Maybe the host BIOS</div>
<div>does strange things in enabling ROMs on the second root bridge, or</div>
<div>doesn't enable them at all. With GeForce cards, you preferably want to</div>
<div>use an OVMF VM or you want to enable x-vga=on with the vfio-pci</div>
<div>device. Thanks,</div>
<div> </div>
<div>Alex</div>
</div></blockquote>
</div></body></html>