<div dir="ltr">Hi,<div><br></div><div>I have a guest enabled vIOMMU, but on the guest there are several devices in the same iommu group.</div><div>Could someone help to check if I missed something? </div><div>Thank you very much!</div><div><br></div><div>1. guest xml:</div><div># virsh edit q</div><div>...</div><div><div><os></div><div> <type arch='x86_64' machine='pc-q35-rhel7.5.0'>hvm</type></div><div> <loader readonly='yes' secure='yes' type='pflash'>/usr/share/OVMF/OVMF_CODE.secboot.fd</loader></div><div> <nvram>/var/lib/libvirt/qemu/nvram/q_VARS.fd</nvram></div><div> </os></div></div><div>...</div><div><div> <features></div><div> ...</div><div> <ioapic driver='qemu'/></div><div> </features></div></div><div><div> <cpu mode='host-passthrough' check='none'></div><div> <feature policy='require' name='vmx'/></div><div> </cpu></div></div><div>...</div><div><devices></div><div>...</div><div><div> <controller type='pci' index='7' model='pcie-root-port'></div><div> <model name='pcie-root-port'/></div><div> <target chassis='7' port='0x15'/></div><div> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/></div><div> </controller></div></div><div><interface type='network'></div><div> <mac address='52:54:00:b9:ff:90'/></div><div> <source network='default'/></div><div> <model type='e1000'/></div><div> <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/></div><div> </interface></div><div><div><iommu model='intel'></div><div> <driver intremap='on' caching_mode='on' iotlb='on'/></div><div> </iommu></div></div><div><span style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"></devices></span><br></div><div>...</div><div>2. guest has 'intel_iommu=on' enabled in kernel cmdline, then reboot guest</div><div><br></div><div>3. log in guest to check:</div><div><div># dmesg | grep -i DMAR</div><div>[ 0.000000] ACPI: DMAR 000000007d83f000 00050 (v01 BOCHS BXPCDMAR 00000001 BXPC 00000001)</div><div>[ 0.000000] DMAR: IOMMU enabled</div><div>[ 0.155178] DMAR: Host address width 39</div><div>[ 0.155180] DMAR: DRHD base: 0x000000fed90000 flags: 0x1</div><div>[ 0.155221] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap 12008c22260286 ecap f00f5e</div><div>[ 0.155228] DMAR: ATSR flags: 0x1</div><div>[ 0.155231] DMAR-IR: IOAPIC id 0 under DRHD base 0xfed90000 IOMMU 0</div><div>[ 0.155232] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.</div><div>[ 0.156843] DMAR-IR: Enabled IRQ remapping in x2apic mode</div><div>[ 2.112369] DMAR: No RMRR found</div><div>[ 2.112505] DMAR: dmar0: Using Queued invalidation</div><div>[ 2.112669] DMAR: Setting RMRR:</div><div>[ 2.112671] DMAR: Prepare 0-16MiB unity mapping for LPC</div><div>[ 2.112820] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]</div><div>[ 2.211577] DMAR: Intel(R) Virtualization Technology for Directed I/O</div></div><div>===> This is expected</div><div><br></div><div><div># dmesg | grep -i iommu |grep device</div><div>[ 2.212267] iommu: Adding device 0000:00:00.0 to group 0</div><div>[ 2.212287] iommu: Adding device 0000:00:01.0 to group 1</div><div>[ 2.212372] iommu: Adding device 0000:00:02.0 to group 2</div><div>[ 2.212392] iommu: Adding device 0000:00:02.1 to group 2</div><div>[ 2.212411] iommu: Adding device 0000:00:02.2 to group 2</div><div>[ 2.212444] iommu: Adding device 0000:00:02.3 to group 2</div><div>[ 2.212464] iommu: Adding device 0000:00:02.4 to group 2</div><div>[ 2.212482] iommu: Adding device 0000:00:02.5 to group 2</div><div>[ 2.212520] iommu: Adding device 0000:00:1d.0 to group 3</div><div>[ 2.212533] iommu: Adding device 0000:00:1d.1 to group 3</div><div>[ 2.212541] iommu: Adding device 0000:00:1d.2 to group 3</div><div>[ 2.212550] iommu: Adding device 0000:00:1d.7 to group 3</div><div>[ 2.212567] iommu: Adding device 0000:00:1f.0 to group 4</div><div>[ 2.212576] iommu: Adding device 0000:00:1f.2 to group 4</div><div>[ 2.212585] iommu: Adding device 0000:00:1f.3 to group 4</div><div>[ 2.212599] iommu: Adding device 0000:01:00.0 to group 2</div><div>[ 2.212605] iommu: Adding device 0000:02:01.0 to group 2</div><div>[ 2.212621] iommu: Adding device 0000:04:00.0 to group 2</div><div>[ 2.212634] iommu: Adding device 0000:05:00.0 to group 2</div><div>[ 2.212646] iommu: Adding device 0000:06:00.0 to group 2</div><div>[ 2.212657] iommu: Adding device 0000:07:00.0 to group 2</div></div><div>====> several devices in the same iommu group</div><div><br></div><div><div># virsh nodedev-dumpxml pci_0000_07_00_0</div><div><device></div><div> <name>pci_0000_07_00_0</name></div><div> <path>/sys/devices/pci0000:00/0000:00:02.5/0000:07:00.0</path></div><div> <parent>pci_0000_00_02_5</parent></div><div> <driver></div><div> <name>e1000</name></div><div> </driver></div><div> <capability type='pci'></div><div> <domain>0</domain></div><div> <bus>7</bus></div><div> <slot>0</slot></div><div> <function>0</function></div><div> <product id='0x100e'>82540EM Gigabit Ethernet Controller</product></div><div> <vendor id='0x8086'>Intel Corporation</vendor></div><div> <iommuGroup number='2'></div><div> <address domain='0x0000' bus='0x00' slot='0x02' function='0x0'/></div><div> <address domain='0x0000' bus='0x00' slot='0x02' function='0x1'/></div><div> <address domain='0x0000' bus='0x00' slot='0x02' function='0x2'/></div><div> <address domain='0x0000' bus='0x00' slot='0x02' function='0x3'/></div><div> <address domain='0x0000' bus='0x00' slot='0x02' function='0x4'/></div><div> <address domain='0x0000' bus='0x00' slot='0x02' function='0x5'/></div><div> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/></div><div> <address domain='0x0000' bus='0x02' slot='0x01' function='0x0'/></div><div> <address domain='0x0000' bus='0x04' slot='0x00' function='0x0'/></div><div> <address domain='0x0000' bus='0x05' slot='0x00' function='0x0'/></div><div> <address domain='0x0000' bus='0x06' slot='0x00' function='0x0'/></div><div> <address domain='0x0000' bus='0x07' slot='0x00' function='0x0'/></div><div> </iommuGroup></div><div> </capability></div><div></device></div></div><div><br></div><div>Thus, can not attach the device to L2 guest:</div><div><div># cat hostdev.xml</div><div><hostdev mode='subsystem' type='pci' managed='yes'></div><div> <source></div><div> <address domain='0x0000' bus='0x07' slot='0x00' function='0x0'/></div><div> </source></div><div> </hostdev></div><div><div># virsh attach-device rhel hostdev.xml</div><div>error: Failed to attach device from hostdev.xml</div><div>error: internal error: unable to execute QEMU command 'device_add': vfio error: 0000:07:00.0: group 2 is not viable</div></div><div><div class="gmail_signature"><br><br>-------<br>Best Regards,<br>Yalan Zhang<br>IRC: yalzhang<br>Internal phone: 8389413</div></div> </div></div>