[vfio-users] VFIO PCI passthrough problems booting VM

Sat Sep 12 13:34:05 UTC 2015

How can we tell if a device might do DMA or not?
For example, it is rather obvious that a hard drive controller would do
DMA, but what about soundcards? Regular(not XDMA crossfire) GPUs?
Why a NIC would need to do DMA?
On Sep 12, 2015 4:04 PM, "Alex Williamson" <alex.williamson at redhat.com>
wrote:

> On Sat, 2015-09-12 at 13:42 +1000, Phill Edwards wrote:
> > I'm running a CentOS 7.1 KVM hypervisor with kernel 3.10.0-229.el7.x86_6
> > and trying to pass through a DVB-T card to a VM. But every time I try to
> > start the VM with virt-manager I get this error:
> >
> > Error starting domain: internal error: process exited while connecting to
> > monitor: 2015-09-10T01:15:53.633408Z qemu-kvm: -device
> > vfio-pci,host=05:02.0,id=hostdev0,bus=pci.0,addr=0x9: vfio: error, group
> 11
> > is not viable, please ensure all devices within the iommu_group are bound
> > to their vfio bus driver.
> > 2015-09-10T01:15:53.633447Z qemu-kvm: -device
> > vfio-pci,host=05:02.0,id=hostdev0,bus=pci.0,addr=0x9: vfio: failed to get
> > group 11
> > 2015-09-10T01:15:53.633463Z qemu-kvm: -device
> > vfio-pci,host=05:02.0,id=hostdev0,bus=pci.0,addr=0x9: Device
> initialization
> > failed.
> > 2015-09-10T01:15:53.633479Z qemu-kvm: -device
> > vfio-pci,host=05:02.0,id=hostdev0,bus=pci.0,addr=0x9: Device 'vfio-pci'
> > could not be initialized
> >
> > I understand the problem is that I have multiple PCI devices in group 11:
> > # find /sys/kernel/iommu_groups/ -type l
> > /sys/kernel/iommu_groups/11/devices/0000:00:1e.0
> > /sys/kernel/iommu_groups/11/devices/0000:05:00.0
> > /sys/kernel/iommu_groups/11/devices/0000:05:00.1
> > /sys/kernel/iommu_groups/11/devices/0000:05:02.0
> > /sys/kernel/iommu_groups/11/devices/0000:05:02.2
>
> > My DVB-T card has 2 PCI IDs: 0000:05:02.0 and 0000:05:02.2. Output of
> lspci
> > command follows:
> >
> > # lspci -nnk
> > 00:1e.0 PCI bridge [0604]: Intel Corporation 82801 PCI Bridge [8086:244e]
> > 05:00.0 Ethernet controller [0200]: Intel Corporation 82546EB Gigabit
> > Ethernet Controller (Copper) [8086:1010] (rev 01)
> >         Subsystem: Compaq Computer Corporation NC7170 Gigabit Server
> > Adapter [0e11:00db]
> >         Kernel driver in use: e1000
> > 05:00.1 Ethernet controller [0200]: Intel Corporation 82546EB Gigabit
> > Ethernet Controller (Copper) [8086:1010] (rev 01)
> >         Subsystem: Compaq Computer Corporation NC7170 Gigabit Server
> > Adapter [0e11:00db]
> >         Kernel driver in use: e1000
> > 05:02.0 Multimedia video controller [0400]: Conexant Systems, Inc.
> > CX23880/1/2/3 PCI Video and Audio Decoder [14f1:8800] (rev 05)
> >         Subsystem: KWorld Computer Co. Ltd. KWorld/VStream XPert DVB-T
> > [17de:08a6]
> > 05:02.2 Multimedia controller [0480]: Conexant Systems, Inc.
> CX23880/1/2/3
> > PCI Video and Audio Decoder [MPEG Port] [14f1:8802] (rev 05)
> >         Subsystem: KWorld Computer Co. Ltd. KWorld/VStream XPert DVB-T
> > [17de:08a6]
> >
> >
> > I've been reading some of the posts at http://vfio.blogspot.com.au/ and
> > added this to GRUB_CMDLINE_LINUX in /etc/sysconfig/grub:
> > iommu=pt intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1
> > pci-stub.ids=14f1:8802,14f1:8800.
> >
> > Unfortunately I still can't get the VM to start so obviously what I added
> > to the kernel command line didn't work.
> >
> > I've also tried adding these options to GRUB_CMDLINE_LINUX but these
> result
> > in the same error when starting the VM:
> > iommu=pt intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1
> > pcie_acs_override=id:14f1:8802
> >
> >    and
> >
> > iommu=pt intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1
> > pcie_acs_override=downstream
> >
> >
> > I thought one of these would fix it but I just cannot get it working. Can
> > anyone please give me some pointers as to what I'm doing wrong here?
>
> The short answer is that you cannot do what you want to do and it's a
> hardware limitation of conventional PCI that prevents it.  The QEMU
> invocation is telling you how to fix it:
>
> "vfio: error, group 11 is not viable, please ensure all devices within
> the iommu_group are bound to their vfio bus driver."
>
> and the blog post about IOMMU groups[1], that I already pointed you to
> in the blog comments, tells you why.  So, you can either choose to bind
> your NICs at 05:00.0 and 05:00.1 to pci-stub or vfio-pci or replace it
> with a PCIe NIC or replace your DVB-T card with a PCIe version or maybe
> get a PCIe-to-PCI adapter and move one of the cards somewhere else.
>
> Conventional PCI does not include a requester ID in bus transactions.
> The requester ID is what the IOMMU uses to lookup the translations for a
> device.  In this case the PCIe-to-PCI bridge (00:1e.0) adds the
> requester ID, which means that the IOMMU cannot differentiate between
> the devices on bus 05.  The ACS override patches will not help you here.
> The old pci-assign driver will let you assign these devices separately,
> but it will fail spectacularly as your NICs suddenly get their DMA
> mappings replaced with the I/O virtual address space of the VM.  Ok, it
> won't be that spectacular, but it won't work.
>
> So to repeat, in the existing configuration, your only option is to give
> up use of the NICs at 05:00.x by binding them to either pci-stub or
> vfio-pci.  Thanks,
>
> Alex
>
> [1] http://vfio.blogspot.com/2014/08/iommu-groups-inside-and-out.html
>
> _______________________________________________
> vfio-users mailing list
> vfio-users at redhat.com
> https://www.redhat.com/mailman/listinfo/vfio-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20150912/f41d5dbb/attachment.htm>