<html><head></head><body><div style="font-family: Verdana;font-size: 12.0px;"><div>
<div><span style="font-family:courier new,courier,monospace;"> Hello,</span>
<div> </div>
<div><span style="font-family:courier new,courier,monospace;">I want to passthrough an InfiniBand-Card to a guest and have some problems with that. My CPU (Intel Xeon E5-2650) is vt-d capable and vt-d is activated in the BIOS. My CentOS 6.4 (2.6.32-358.11.1.el6.x86_64) is booting with the command line "... intel_iommu=on iommu=pt". </span></div>
<div> </div>
<div><span style="font-family:courier new,courier,monospace;">The guest has the same OS and kernel as the host, also the driver used for the InfiniBand-Card is the same (used the official Mellanox OFED driver pack).</span></div>
<div> </div>
<div><span style="font-family:courier new,courier,monospace;">This is what I've done:</span></div>
<div> </div>
<div><span style="font-family:courier new,courier,monospace;">0.) Verify that InfiniBand working on host:</span></div>
<div>
<div><span style="font-family:courier new,courier,monospace;">root@host# ibv_devinfo<br/>
hca_id: mthca0<br/>
transport: InfiniBand (0)<br/>
fw_ver: 1.0.800<br/>
node_guid: ...<br/>
sys_image_guid: ...<br/>
vendor_id: 0x08f1<br/>
vendor_part_id: 25204<br/>
hw_ver: 0xA0<br/>
board_id: ...<br/>
phys_port_cnt: 1<br/>
port: 1<br/>
state: PORT_ACTIVE (4)<br/>
max_mtu: 2048 (4)<br/>
active_mtu: 2048 (4)<br/>
sm_lid: 3<br/>
port_lid: 5<br/>
port_lmc: 0x00<br/>
link_layer: InfiniBand</span></div>
<div> </div>
<div><span style="font-family:courier new,courier,monospace;">1.) DMAR and IOMMU messages at host:</span></div>
<div>
<div><span style="font-family:courier new,courier,monospace;">root@host# dmesg | grep -e DMAR -e IOMMU<br/>
ACPI: DMAR 000000007e27ea30 00160 (v01 A M I OEMDMAR 00000001 INTL 00000001)<br/>
Intel-IOMMU: enabled<br/>
dmar: IOMMU 0: reg_base_addr fbffe000 ver 1:0 cap d2078c106f0462 ecap f020fe<br/>
dmar: IOMMU 1: reg_base_addr dfffc000 ver 1:0 cap d2078c106f0462 ecap f020fe<br/>
IOMMU 0xfbffe000: using Queued invalidation<br/>
IOMMU 0xdfffc000: using Queued invalidation<br/>
IOMMU: hardware identity mapping for device 0000:00:00.0</span></div>
<div><span style="font-family:courier new,courier,monospace;">[... (a lot of mapping messages) ...]</span></div>
<div><span style="font-family:courier new,courier,monospace;">IOMMU: hardware identity mapping for device 0000:81:00.0 (this is the IB card)<br/>
IOMMU: Setting RMRR:<br/>
IOMMU: Prepare 0-16MiB unity mapping for LPC<br/>
dmar: DMAR:[DMA Read] Request device [81:00.0] fault addr 107294f000<br/>
DMAR:[fault reason 06] PTE Read access is not set<br/>
dmar: DMAR:[DMA Read] Request device [81:00.0] fault addr 107294f000<br/>
DMAR:[fault reason 06] PTE Read access is not set</span></div>
</div>
</div>
<div><span style="font-family:courier new,courier,monospace;">2.) Output of lspci at host</span></div>
<div><span style="font-family:courier new,courier,monospace;">81:00.0 InfiniBand: Mellanox Technologies MT25204 [InfiniHost III Lx HCA] (rev a0)<br/>
Subsystem: Mellanox Technologies MT25204 [InfiniHost III Lx HCA]<br/>
Flags: fast devsel, IRQ 114<br/>
Memory at f8a00000 (64-bit, non-prefetchable) [size=1M]<br/>
Memory at 3c1e00000000 (64-bit, prefetchable) [size=8M]<br/>
Capabilities: [40] Power Management version 2<br/>
Capabilities: [48] Vital Product Data<br/>
Capabilities: [90] MSI: Enable- Count=1/32 Maskable- 64bit+<br/>
Capabilities: [84] MSI-X: Enable- Count=32 Masked-<br/>
Capabilities: [60] Express Endpoint, MSI 00</span></div>
<div><span style="font-family:courier new,courier,monospace;"> Kernel driver in use: ib_mthca<br/>
Kernel modules: ib_mthca</span></div>
<div> </div>
<div><span style="font-family:courier new,courier,monospace;">3.) Adding the device with virsh edit:<br/>
<hostdev mode='subsystem' type='pci' managed='yes'><br/>
<source><br/>
<address domain='0x0000' bus='0x81' slot='0x00' function='0x0'/><br/>
</source><br/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/><br/>
</hostdev></span></div>
<div>
<div> </div>
<div><span style="font-family:courier new,courier,monospace;">The last <address .../>-Tag with "slot=0x07" was added automatically by virsh.</span></div>
<div> </div>
<div><span style="font-family:courier new,courier,monospace;">4.) Removing device with nodedev-dettach:</span></div>
<div>
<div><span style="font-family:courier new,courier,monospace;">root@host# virsh nodedev-dettach pci_0000_81_00_0<br/>
Device pci_0000_81_00_0 detached</span></div>
<div> </div>
<div><span style="font-family:courier new,courier,monospace;">lspci on the host now shows the same as in (1) but the "Kernel driver in use" changed:</span></div>
</div>
<div><span style="font-family:courier new,courier,monospace;"> Kernel driver in use: pci-stub</span></div>
<div> </div>
<div><span style="font-family:courier new,courier,monospace;">5.) InfiniBand now stops working on host (as expected):</span></div>
<div>
<div><span style="font-family:courier new,courier,monospace;">root@host# ibv_devinfo<br/>
No IB devices found</span></div>
<div> </div>
<div><span style="font-family:courier new,courier,monospace;">6.) starting guest, then lspci -v:</span></div>
<div>
<div><span style="font-family:courier new,courier,monospace;">00:07.0 InfiniBand: Mellanox Technologies MT25204 [InfiniHost III Lx HCA] (rev a0)<br/>
Subsystem: Mellanox Technologies MT25204 [InfiniHost III Lx HCA]<br/>
Physical Slot: 7<br/>
Flags: fast devsel, IRQ 10<br/>
Memory at f2100000 (32-bit, non-prefetchable) [size=1M]<br/>
Memory at f2800000 (32-bit, prefetchable) [size=8M]<br/>
Capabilities: [48] Vital Product Data<br/>
Capabilities: [60] Express Endpoint, MSI 00<br/>
Capabilities: [40] Power Management version 2<br/>
Capabilities: [84] MSI-X: Enable- Count=32 Masked-<br/>
Capabilities: [90] MSI: Enable- Count=1/32 Maskable- 64bit-<br/>
Kernel modules: ib_mthca</span></div>
<div> </div>
<div><span style="font-family:courier new,courier,monospace;">I noticed that there is a difference: The memory on guest is 32-bit, on the host it says 64-bit</span></div>
<div> </div>
<div><span style="font-family:courier new,courier,monospace;">7.) IB not working on guest:</span></div>
<div><span style="font-family:courier new,courier,monospace;">root@guest# ibv_devinfo<br/>
No IB devices found</span></div>
<div><span style="font-family:courier new,courier,monospace;">root@guest# ibhosts</span></div>
<div><span style="font-family:courier new,courier,monospace;">src/query_smp.c:228; can't open UMAD port ((null):0)<br/>
/usr/sbin/ibnetdiscover: iberror: failed: discover failed</span></div>
</div>
</div>
<div> </div>
<div><span style="font-family:courier new,courier,monospace;">Do you have any clue where I must tweak my settings in order to get InfiniBand working on my virtual machine? If this is no libvirt problem, I am sorry :)</span></div>
<div> </div>
<div><span style="font-family:courier new,courier,monospace;">Regards,</span></div>
<div><span style="font-family:courier new,courier,monospace;">Sebastian</span></div>
</div>
</div>
</div></div></body></html>