[libvirt-users] multiple devices in the same iommu group in L1 guest

[Yalan Zhang]

Hi,

I have a guest enabled vIOMMU, but on the guest there are several devices
in the same iommu group.
Could someone help to check if I missed something?
Thank you very much!

1. guest xml:
# virsh edit q
...
<os>
    <type arch='x86_64' machine='pc-q35-rhel7.5.0'>hvm</type>
    <loader readonly='yes' secure='yes'
type='pflash'>/usr/share/OVMF/OVMF_CODE.secboot.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/q_VARS.fd</nvram>
  </os>
...
 <features>
 ...
    <ioapic driver='qemu'/>
  </features>
 <cpu mode='host-passthrough' check='none'>
    <feature policy='require' name='vmx'/>
  </cpu>
...
<devices>
...
 <controller type='pci' index='7' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='7' port='0x15'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
function='0x5'/>
    </controller>
<interface type='network'>
      <mac address='52:54:00:b9:ff:90'/>
      <source network='default'/>
      <model type='e1000'/>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00'
function='0x0'/>
    </interface>
<iommu model='intel'>
      <driver intremap='on' caching_mode='on' iotlb='on'/>
    </iommu>
</devices>
...
2. guest has 'intel_iommu=on' enabled in kernel cmdline, then reboot guest

3. log in guest to check:
# dmesg  | grep -i DMAR
[    0.000000] ACPI: DMAR 000000007d83f000 00050 (v01 BOCHS  BXPCDMAR
00000001 BXPC 00000001)
[    0.000000] DMAR: IOMMU enabled
[    0.155178] DMAR: Host address width 39
[    0.155180] DMAR: DRHD base: 0x000000fed90000 flags: 0x1
[    0.155221] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap
12008c22260286 ecap f00f5e
[    0.155228] DMAR: ATSR flags: 0x1
[    0.155231] DMAR-IR: IOAPIC id 0 under DRHD base  0xfed90000 IOMMU 0
[    0.155232] DMAR-IR: Queued invalidation will be enabled to support
x2apic and Intr-remapping.
[    0.156843] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    2.112369] DMAR: No RMRR found
[    2.112505] DMAR: dmar0: Using Queued invalidation
[    2.112669] DMAR: Setting RMRR:
[    2.112671] DMAR: Prepare 0-16MiB unity mapping for LPC
[    2.112820] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 -
0xffffff]
[    2.211577] DMAR: Intel(R) Virtualization Technology for Directed I/O
===> This is expected

# dmesg  | grep -i iommu  |grep device
[    2.212267] iommu: Adding device 0000:00:00.0 to group 0
[    2.212287] iommu: Adding device 0000:00:01.0 to group 1
[    2.212372] iommu: Adding device 0000:00:02.0 to group 2
[    2.212392] iommu: Adding device 0000:00:02.1 to group 2
[    2.212411] iommu: Adding device 0000:00:02.2 to group 2
[    2.212444] iommu: Adding device 0000:00:02.3 to group 2
[    2.212464] iommu: Adding device 0000:00:02.4 to group 2
[    2.212482] iommu: Adding device 0000:00:02.5 to group 2
[    2.212520] iommu: Adding device 0000:00:1d.0 to group 3
[    2.212533] iommu: Adding device 0000:00:1d.1 to group 3
[    2.212541] iommu: Adding device 0000:00:1d.2 to group 3
[    2.212550] iommu: Adding device 0000:00:1d.7 to group 3
[    2.212567] iommu: Adding device 0000:00:1f.0 to group 4
[    2.212576] iommu: Adding device 0000:00:1f.2 to group 4
[    2.212585] iommu: Adding device 0000:00:1f.3 to group 4
[    2.212599] iommu: Adding device 0000:01:00.0 to group 2
[    2.212605] iommu: Adding device 0000:02:01.0 to group 2
[    2.212621] iommu: Adding device 0000:04:00.0 to group 2
[    2.212634] iommu: Adding device 0000:05:00.0 to group 2
[    2.212646] iommu: Adding device 0000:06:00.0 to group 2
[    2.212657] iommu: Adding device 0000:07:00.0 to group 2
====> several devices in the same iommu group

# virsh nodedev-dumpxml pci_0000_07_00_0
<device>
  <name>pci_0000_07_00_0</name>
  <path>/sys/devices/pci0000:00/0000:00:02.5/0000:07:00.0</path>
  <parent>pci_0000_00_02_5</parent>
  <driver>
    <name>e1000</name>
  </driver>
  <capability type='pci'>
    <domain>0</domain>
    <bus>7</bus>
    <slot>0</slot>
    <function>0</function>
    <product id='0x100e'>82540EM Gigabit Ethernet Controller</product>
    <vendor id='0x8086'>Intel Corporation</vendor>
    <iommuGroup number='2'>
      <address domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
      <address domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
      <address domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
      <address domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
      <address domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
      <address domain='0x0000' bus='0x00' slot='0x02' function='0x5'/>
      <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      <address domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>
      <address domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
      <address domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
      <address domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
      <address domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </iommuGroup>
  </capability>
</device>

Thus, can not attach the device to L2 guest:
# cat hostdev.xml
<hostdev mode='subsystem' type='pci' managed='yes'>
    <source>
      <address domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </source>
  </hostdev>
# virsh attach-device rhel hostdev.xml
error: Failed to attach device from hostdev.xml
error: internal error: unable to execute QEMU command 'device_add': vfio
error: 0000:07:00.0: group 2 is not viable


-------
Best Regards,
Yalan Zhang
IRC: yalzhang
Internal phone: 8389413
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/libvirt-users/attachments/20180703/2f3d4229/attachment.htm>