[libvirt-users] passthrough of PCI-device
Weis, Michael (DWIE)
michael.weis at smithsdetection.com
Tue Oct 14 06:20:49 UTC 2014
I enabled iommu in the grub-config:
grep iommu /boot/grub2/grub.cfg
linux16 /vmlinuz-3.10.0-123.8.1.el7.x86_64 root=UUID=9cfb58ce-b401-47d1-8dc0-babbd1f21ec3 ro rootflags=subvol=root crashkernel=auto vconsole.keymap=de vconsole.font=latarcyrheb-sun16 rhgb quiet LANG=de_DE.UTF-8 intel_iommu=on
Here is the relevant output of /var/log/messages during boot:
Oct 7 09:18:01 artemis2 kernel: dmar: Host address width 39
Oct 7 09:18:01 artemis2 kernel: dmar: DRHD base: 0x000000fed90000 flags: 0x1
Oct 7 09:18:01 artemis2 kernel: dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap d2008c20660462 ecap f010da
Oct 7 09:18:01 artemis2 kernel: dmar: RMRR base: 0x000000ceefa000 end: 0x000000cef06fff
Oct 7 09:18:01 artemis2 kernel: IOAPIC id 2 under DRHD base 0xfed90000 IOMMU 0
Later on I get the following error:
Oct 7 09:18:01 artemis2 kernel: DMAR: No ATSR found
Oct 7 09:18:01 artemis2 kernel: IOMMU 0 0xfed90000: using Queued invalidation
Oct 7 09:18:01 artemis2 kernel: IOMMU: Setting RMRR:
Oct 7 09:18:01 artemis2 kernel: IOMMU: Setting identity map for device 0000:00:1d.0 [0xceefa000 - 0xcef06fff]
Oct 7 09:18:01 artemis2 kernel: IOMMU: Setting identity map for device 0000:00:1a.0 [0xceefa000 - 0xcef06fff]
Oct 7 09:18:01 artemis2 kernel: IOMMU: Setting identity map for device 0000:00:14.0 [0xceefa000 - 0xcef06fff]
Oct 7 09:18:01 artemis2 kernel: IOMMU: Prepare 0-16MiB unity mapping for LPC
Oct 7 09:18:01 artemis2 kernel: IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
Oct 7 09:18:01 artemis2 kernel: PCI-DMA: Intel(R) Virtualization Technology for Directed I/O
Oct 7 09:18:01 artemis2 kernel: ------------[ cut here ]------------
Oct 7 09:18:01 artemis2 kernel: WARNING: at drivers/pci/search.c:46 pci_find_upstream_pcie_bridge+0x87/0x90()
Oct 7 09:18:01 artemis2 kernel: Modules linked in:
Oct 7 09:18:01 artemis2 kernel: CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.10.0-123.8.1.el7.x86_64 #1
Oct 7 09:18:01 artemis2 kernel: Hardware name: Hewlett-Packard HP Z230 Tower Workstation/1905, BIOS L51 v01.13 10/18/2013
Oct 7 09:18:01 artemis2 kernel: 0000000000000000 00000000eccfd72f ffff8802220e3cf0 ffffffff815e237b
Oct 7 09:18:01 artemis2 kernel: ffff8802220e3d28 ffffffff8105dee1 ffff880221f95000 ffff880221f9f000
Oct 7 09:18:01 artemis2 kernel: ffff880221f9f098 0000000000000000 0000000000000000 ffff8802220e3d38
Oct 7 09:18:01 artemis2 kernel: Call Trace:
Oct 7 09:18:01 artemis2 kernel: [<ffffffff815e237b>] dump_stack+0x19/0x1b
Oct 7 09:18:01 artemis2 kernel: [<ffffffff8105dee1>] warn_slowpath_common+0x61/0x80
Oct 7 09:18:01 artemis2 kernel: [<ffffffff8105e00a>] warn_slowpath_null+0x1a/0x20
Oct 7 09:18:01 artemis2 kernel: [<ffffffff812ec607>] pci_find_upstream_pcie_bridge+0x87/0x90
Oct 7 09:18:01 artemis2 kernel: [<ffffffff814aa04d>] intel_iommu_add_device+0x4d/0x230
Oct 7 09:18:01 artemis2 kernel: [<ffffffff814a10f0>] ? bus_set_iommu+0x50/0x50
Oct 7 09:18:01 artemis2 kernel: [<ffffffff814a111a>] add_iommu_group+0x2a/0x50
Oct 7 09:18:01 artemis2 kernel: [<ffffffff813b4443>] bus_for_each_dev+0x73/0xc0
Oct 7 09:18:01 artemis2 kernel: [<ffffffff814a10e8>] bus_set_iommu+0x48/0x50
Oct 7 09:18:01 artemis2 kernel: [<ffffffff81a5837d>] intel_iommu_init+0x3ea/0x413
Oct 7 09:18:01 artemis2 kernel: [<ffffffff81a11c93>] ? memblock_find_dma_reserve+0x147/0x147
Oct 7 09:18:01 artemis2 kernel: [<ffffffff81a11ca5>] pci_iommu_init+0x12/0x3c
Oct 7 09:18:01 artemis2 kernel: [<ffffffff810020e2>] do_one_initcall+0xe2/0x190
Oct 7 09:18:01 artemis2 kernel: [<ffffffff81a09153>] kernel_init_freeable+0x18b/0x22a
Oct 7 09:18:01 artemis2 kernel: [<ffffffff81a0892b>] ? do_early_param+0x88/0x88
Oct 7 09:18:01 artemis2 kernel: [<ffffffff815c3c20>] ? rest_init+0x80/0x80
Oct 7 09:18:01 artemis2 kernel: [<ffffffff815c3c2e>] kernel_init+0xe/0x180
Oct 7 09:18:01 artemis2 kernel: [<ffffffff815f29ac>] ret_from_fork+0x7c/0xb0
Oct 7 09:18:01 artemis2 kernel: [<ffffffff815c3c20>] ? rest_init+0x80/0x80
Oct 7 09:18:01 artemis2 kernel: ---[ end trace 56f25a222bb09abf ]---
Best regards
Michael
-----Ursprüngliche Nachricht-----
Von: libvirt-users-bounces at redhat.com [mailto:libvirt-users-bounces at redhat.com] Im Auftrag von Pierre Schweitzer
Gesendet: Montag, 13. Oktober 2014 13:19
An: libvirt-users at redhat.com
Betreff: Re: [libvirt-users] passthrough of PCI-device
And what about IO MMU ?
On 10/13/2014 12:02 PM, Weis, Michael (DWIE) wrote:
> Hi Pierre,
>
> thanks for your reply.
>
> I am using kernel 3.10.0-123.8.1.el7.x86_64.
> The kernel modul used after nodedev-detach is vfio-pci
>
> This is the output of lspci -vv after I did a virsh nodedev-detatch
> pci_0000_02_00_0
>
> 02:00.0 Ethernet controller: PLX Technology, Inc. Device 235e
> Subsystem: PLX Technology, Inc. Device 235e
> Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
> Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> Interrupt: pin A routed to IRQ 16
> Region 0: Memory at f0810000 (32-bit, non-prefetchable) [disabled] [size=256]
> Region 1: I/O ports at e000 [disabled] [size=256]
> Region 2: Memory at e0000000 (32-bit, non-prefetchable) [disabled] [size=256M]
> Region 3: Memory at f0000000 (32-bit, non-prefetchable) [disabled] [size=8M]
> Expansion ROM at f0800000 [disabled] [size=64K]
> Capabilities: [40] Power Management version 1
> Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
> Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
> Capabilities: [48] CompactPCI hot-swap <?>
> Capabilities: [4c] Vital Product Data
> Unknown large resource type 35, will not decode more.
> Kernel driver in use: vfio-pci
>
>
> Best regards
>
> Michael
>
>
>
>
> -----Ursprüngliche Nachricht-----
> Von: libvirt-users-bounces at redhat.com
> [mailto:libvirt-users-bounces at redhat.com] Im Auftrag von Pierre
> Schweitzer
> Gesendet: Montag, 13. Oktober 2014 10:04
> An: libvirt-users at redhat.com
> Betreff: Re: [libvirt-users] passthrough of PCI-device
>
> Dear Michael,
>
> Did you activate the Intel IO MMU (or its equivalent for AMD)?
> Also, did you load the pci_stub module for Linux? It is mandatory (it will replace current driver for your passed through hardware).
>
> Cheers,
> Pierre
>
> On 10/13/2014 07:54 AM, Weis, Michael (DWIE) wrote:
>> Good morning,
>>
>> there is a typo in my description;
>> the line
>> <address domain='0x0' bus='0x1' slot='0x00' function='0x0'/>
>> should be
>> <address domain='0x0' bus='0x2' slot='0x00' function='0x0'/>
>> That was correct in my xml-file.
>>
>> Isn't there anybody how can help me with that?
>>
>>
>> Regards
>>
>> Michael Weis
>>
>>
>>
>> Von: Weis, Michael (DWIE)
>> Gesendet: Dienstag, 7. Oktober 2014 11:45
>> An: 'libvirt-users at redhat.com'
>> Betreff: passthrough of PCI-device
>>
>> Hello,
>>
>> I try to passthrough a PCI-card to a VM named testvm I want to do
>> that with an xml-file named hga.xml including the following content:
>>
>> <hostdev mode='subsystem' type='pci' managed='yes'>
>> <source>
>> <address domain='0x0' bus='0x1' slot='0x00' function='0x0'/>
>> </source>
>> </hostdev>
>>
>> When I execute
>> virsh attach-device testvm hga.xml
>> I get the following error:
>> Fehler: Fehler beim Anhängen des Geräts von hga.xml
>> (error: error while attaching device from hga.xml)
>> Fehler: Interner Fehler: Kann Qemu-Befehl 'device_add' nicht ausführen: Device initialization failed.
>> (error: internal error: cannot execute 'device_add':
>>
>> In the syslog of the host I get the following errors:
>> Oct 7 11:32:34 artemis2 journal: Interner Fehler: Kann Qemu-Befehl 'device_add' nicht ausführen: Device initialization failed.
>> Oct 7 11:32:53 artemis2 kernel: ffff880220fadd10 ffffffff810f924a
>> ffff880220fadd28 0000000000000246 Oct 7 11:32:53 artemis2 kernel:
>> ffff8800bd07a400 000000007265756d ffffffffa0866ee0 0000000000000000 Oct 7 11:32:53 artemis2 kernel: Call Trace:
>> Oct 7 11:32:53 artemis2 kernel: [<ffffffff815e237b>]
>> dump_stack+0x19/0x1b Oct 7 11:32:53 artemis2 kernel:
>> [<ffffffff810f924a>] __setup_irq+0x54a/0x570 Oct 7 11:32:53 artemis2
>> kernel: [<ffffffffa0866ee0>] ? virqfd_wakeup+0xc0/0xc0 [vfio_pci] Oct
>> 7 11:32:53 artemis2 kernel: [<ffffffff810f93cc>]
>> request_threaded_irq+0xcc/0x170 Oct 7 11:32:53 artemis2 kernel:
>> [<ffffffffa0866b8a>] vfio_intx_set_signal+0x10a/0x1f0 [vfio_pci] Oct
>> 7 11:32:53 artemis2 kernel: [<ffffffffa08670e8>]
>> vfio_pci_set_intx_trigger+0xc8/0x1b0 [vfio_pci] Oct 7 11:32:53
>> artemis2 kernel: [<ffffffffa086799b>]
>> vfio_pci_set_irqs_ioctl+0x7b/0x100 [vfio_pci] Oct 7 11:32:53
>> artemis2
>> kernel: [<ffffffff8115d352>] ? memdup_user+0x42/0x70 Oct 7 11:32:53
>> artemis2 kernel: [<ffffffffa0865e74>] vfio_pci_ioctl+0x314/0xa00
>> [vfio_pci] Oct 7 11:32:53 artemis2 kernel: [<ffffffff81194b6a>] ?
>> kmem_cache_alloc+0x1ba/0x1d0 Oct 7 11:32:53 artemis2 kernel:
>> [<ffffffff811b156c>] ? get_empty_filp+0x5c/0x190 Oct 7 11:32:53
>> artemis2 kernel: [<ffffffff811f1891>] ? fsnotify+0x251/0x350 Oct 7
>> 11:32:53 artemis2 kernel: [<ffffffff81251806>] ?
>> security_file_alloc+0x16/0x20 Oct 7 11:32:53 artemis2 kernel:
>> [<ffffffff811b15a0>] ? get_empty_filp+0x90/0x190 Oct 7 11:32:53 artemis2 kernel: [<ffffffffa084f163>] vfio_device_fops_unl_ioctl+0x23/0x30 [vfio] Oct 7 11:32:53 artemis2 kernel: [<ffffffff811c2af5>] do_vfs_ioctl+0x2e5/0x4c0 Oct 7 11:32:53 artemis2 kernel: [<ffffffff811af780>] ? vfs_write+0x160/0x1e0 Oct 7 11:32:53 artemis2 kernel: [<ffffffff811c2d71>] SyS_ioctl+0xa1/0xc0 Oct 7 11:32:53 artemis2 kernel: [<ffffffff815f2a59>] system_call_fastpath+0x16/0x1b Oct 7 11:32:53 artemis2 journal: Interner Fehler: Kann Qemu-Befehl 'device_add' nicht ausführen: Device initialization failed.
>>
>> My version-information:
>> OS: CentOS Linux release 7.0.1406 (Core)
>> Kernel: Linux artemis2 3.10.0-123.8.1.el7.x86_64
>> libvirt-daemon-driver-secret-1.2.8-1.el7.centos.x86_64
>> libvirt-1.2.8-1.el7.centos.x86_64
>> libvirt-daemon-kvm-1.2.8-1.el7.centos.x86_64
>> libvirt-client-1.2.8-1.el7.centos.x86_64
>> libvirt-daemon-driver-qemu-1.2.8-1.el7.centos.x86_64
>> libvirt-daemon-config-network-1.2.8-1.el7.centos.x86_64
>> libvirt-daemon-driver-interface-1.2.8-1.el7.centos.x86_64
>> libvirt-daemon-1.2.8-1.el7.centos.x86_64
>> libvirt-daemon-driver-nodedev-1.2.8-1.el7.centos.x86_64
>> libvirt-daemon-driver-storage-1.2.8-1.el7.centos.x86_64
>> libvirt-daemon-config-nwfilter-1.2.8-1.el7.centos.x86_64
>> libvirt-daemon-driver-network-1.2.8-1.el7.centos.x86_64
>> libvirt-daemon-driver-lxc-1.2.8-1.el7.centos.x86_64
>> libvirt-daemon-driver-nwfilter-1.2.8-1.el7.centos.x86_64
>>
>> The PCI-Card I want to passthrough:
>> 02:00.0 Ethernet controller: PLX Technology, Inc. Device 235e
>> Subsystem: PLX Technology, Inc. Device 235e
>> Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
>> Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
>> Interrupt: pin A routed to IRQ 16
>> Region 0: Memory at f0810000 (32-bit, non-prefetchable) [disabled] [size=256]
>> Region 1: I/O ports at e000 [disabled] [size=256]
>> Region 2: Memory at e0000000 (32-bit, non-prefetchable) [disabled] [size=256M]
>> Region 3: Memory at f0000000 (32-bit, non-prefetchable) [disabled] [size=8M]
>> Expansion ROM at f0800000 [disabled] [size=64K]
>> Capabilities: [40] Power Management version 1
>> Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
>> Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
>> Capabilities: [48] CompactPCI hot-swap <?>
>> Capabilities: [4c] Vital Product Data
>> Unknown large resource type 35, will not decode more.
>> Kernel driver in use: vfio-pci
>>
>> Please help.
>>
>>
>> Best regards
>>
>> Michael Weis
>>
>>
>>
>> PROPRIETARY: This e-mail contains proprietary information some or all of which may be legally privileged. It is intended for the recipient only. If an addressing or transmission error has misdirected this e-mail, please notify the authority by replying to this e-mail. If you are not the intended recipient you must not use, disclose, distribute, copy, print, or rely on this e-mail.
>>
>>
>>
>> _______________________________________________
>> libvirt-users mailing list
>> libvirt-users at redhat.com
>> https://www.redhat.com/mailman/listinfo/libvirt-users
>>
>
>
> --
> Pierre Schweitzer <pierre at reactos.org> System & Network Administrator
> Senior Kernel Developer ReactOS Deutschland e.V.
>
>
> ______________________________________________________________________
> CAUTION: This message was sent via the Public Internet and its authenticity cannot be guaranteed.
>
> PROPRIETARY: This e-mail contains proprietary information some or all of which may be legally privileged. It is intended for the recipient only. If an addressing or transmission error has misdirected this e-mail, please notify the authority by replying to this e-mail. If you are not the intended recipient you must not use, disclose, distribute, copy, print, or rely on this e-mail.
>
> _______________________________________________
> libvirt-users mailing list
> libvirt-users at redhat.com
> https://www.redhat.com/mailman/listinfo/libvirt-users
>
--
Pierre Schweitzer <pierre at reactos.org>
System & Network Administrator
Senior Kernel Developer
ReactOS Deutschland e.V.
______________________________________________________________________
CAUTION: This message was sent via the Public Internet and its authenticity cannot be guaranteed.
PROPRIETARY: This e-mail contains proprietary information some or all of which may be legally privileged. It is intended for the recipient only. If an addressing or transmission error has misdirected this e-mail, please notify the authority by replying to this e-mail. If you are not the intended recipient you must not use, disclose, distribute, copy, print, or rely on this e-mail.
More information about the libvirt-users
mailing list