[vfio-users] VFIO PCI passthrough problems booting VM

Sat Sep 12 13:43:41 UTC 2015

On Sat, 2015-09-12 at 16:34 +0300, Blank Field wrote:
> How can we tell if a device might do DMA or not?
> For example, it is rather obvious that a hard drive controller would do
> DMA, but what about soundcards? Regular(not XDMA crossfire) GPUs?
> Why a NIC would need to do DMA?

https://en.wikipedia.org/wiki/Direct_memory_access

"Many hardware systems use DMA, including disk drive controllers,
graphics cards, network cards and sound cards."

On a PC, if it has a PCI or PCIe connector, it's capable of DMA and
likely uses it.

> On Sep 12, 2015 4:04 PM, "Alex Williamson" <alex.williamson at redhat.com>
> wrote:
> 
> > On Sat, 2015-09-12 at 13:42 +1000, Phill Edwards wrote:
> > > I'm running a CentOS 7.1 KVM hypervisor with kernel 3.10.0-229.el7.x86_6
> > > and trying to pass through a DVB-T card to a VM. But every time I try to
> > > start the VM with virt-manager I get this error:
> > >
> > > Error starting domain: internal error: process exited while connecting to
> > > monitor: 2015-09-10T01:15:53.633408Z qemu-kvm: -device
> > > vfio-pci,host=05:02.0,id=hostdev0,bus=pci.0,addr=0x9: vfio: error, group
> > 11
> > > is not viable, please ensure all devices within the iommu_group are bound
> > > to their vfio bus driver.
> > > 2015-09-10T01:15:53.633447Z qemu-kvm: -device
> > > vfio-pci,host=05:02.0,id=hostdev0,bus=pci.0,addr=0x9: vfio: failed to get
> > > group 11
> > > 2015-09-10T01:15:53.633463Z qemu-kvm: -device
> > > vfio-pci,host=05:02.0,id=hostdev0,bus=pci.0,addr=0x9: Device
> > initialization
> > > failed.
> > > 2015-09-10T01:15:53.633479Z qemu-kvm: -device
> > > vfio-pci,host=05:02.0,id=hostdev0,bus=pci.0,addr=0x9: Device 'vfio-pci'
> > > could not be initialized
> > >
> > > I understand the problem is that I have multiple PCI devices in group 11:
> > > # find /sys/kernel/iommu_groups/ -type l
> > > /sys/kernel/iommu_groups/11/devices/0000:00:1e.0
> > > /sys/kernel/iommu_groups/11/devices/0000:05:00.0
> > > /sys/kernel/iommu_groups/11/devices/0000:05:00.1
> > > /sys/kernel/iommu_groups/11/devices/0000:05:02.0
> > > /sys/kernel/iommu_groups/11/devices/0000:05:02.2
> >
> > > My DVB-T card has 2 PCI IDs: 0000:05:02.0 and 0000:05:02.2. Output of
> > lspci
> > > command follows:
> > >
> > > # lspci -nnk
> > > 00:1e.0 PCI bridge [0604]: Intel Corporation 82801 PCI Bridge [8086:244e]
> > > 05:00.0 Ethernet controller [0200]: Intel Corporation 82546EB Gigabit
> > > Ethernet Controller (Copper) [8086:1010] (rev 01)
> > >         Subsystem: Compaq Computer Corporation NC7170 Gigabit Server
> > > Adapter [0e11:00db]
> > >         Kernel driver in use: e1000
> > > 05:00.1 Ethernet controller [0200]: Intel Corporation 82546EB Gigabit
> > > Ethernet Controller (Copper) [8086:1010] (rev 01)
> > >         Subsystem: Compaq Computer Corporation NC7170 Gigabit Server
> > > Adapter [0e11:00db]
> > >         Kernel driver in use: e1000
> > > 05:02.0 Multimedia video controller [0400]: Conexant Systems, Inc.
> > > CX23880/1/2/3 PCI Video and Audio Decoder [14f1:8800] (rev 05)
> > >         Subsystem: KWorld Computer Co. Ltd. KWorld/VStream XPert DVB-T
> > > [17de:08a6]
> > > 05:02.2 Multimedia controller [0480]: Conexant Systems, Inc.
> > CX23880/1/2/3
> > > PCI Video and Audio Decoder [MPEG Port] [14f1:8802] (rev 05)
> > >         Subsystem: KWorld Computer Co. Ltd. KWorld/VStream XPert DVB-T
> > > [17de:08a6]
> > >
> > >
> > > I've been reading some of the posts at http://vfio.blogspot.com.au/ and
> > > added this to GRUB_CMDLINE_LINUX in /etc/sysconfig/grub:
> > > iommu=pt intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1
> > > pci-stub.ids=14f1:8802,14f1:8800.
> > >
> > > Unfortunately I still can't get the VM to start so obviously what I added
> > > to the kernel command line didn't work.
> > >
> > > I've also tried adding these options to GRUB_CMDLINE_LINUX but these
> > result
> > > in the same error when starting the VM:
> > > iommu=pt intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1
> > > pcie_acs_override=id:14f1:8802
> > >
> > >    and
> > >
> > > iommu=pt intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1
> > > pcie_acs_override=downstream
> > >
> > >
> > > I thought one of these would fix it but I just cannot get it working. Can
> > > anyone please give me some pointers as to what I'm doing wrong here?
> >
> > The short answer is that you cannot do what you want to do and it's a
> > hardware limitation of conventional PCI that prevents it.  The QEMU
> > invocation is telling you how to fix it:
> >
> > "vfio: error, group 11 is not viable, please ensure all devices within
> > the iommu_group are bound to their vfio bus driver."
> >
> > and the blog post about IOMMU groups[1], that I already pointed you to
> > in the blog comments, tells you why.  So, you can either choose to bind
> > your NICs at 05:00.0 and 05:00.1 to pci-stub or vfio-pci or replace it
> > with a PCIe NIC or replace your DVB-T card with a PCIe version or maybe
> > get a PCIe-to-PCI adapter and move one of the cards somewhere else.
> >
> > Conventional PCI does not include a requester ID in bus transactions.
> > The requester ID is what the IOMMU uses to lookup the translations for a
> > device.  In this case the PCIe-to-PCI bridge (00:1e.0) adds the
> > requester ID, which means that the IOMMU cannot differentiate between
> > the devices on bus 05.  The ACS override patches will not help you here.
> > The old pci-assign driver will let you assign these devices separately,
> > but it will fail spectacularly as your NICs suddenly get their DMA
> > mappings replaced with the I/O virtual address space of the VM.  Ok, it
> > won't be that spectacular, but it won't work.
> >
> > So to repeat, in the existing configuration, your only option is to give
> > up use of the NICs at 05:00.x by binding them to either pci-stub or
> > vfio-pci.  Thanks,
> >
> > Alex
> >
> > [1] http://vfio.blogspot.com/2014/08/iommu-groups-inside-and-out.html
> >
> > _______________________________________________
> > vfio-users mailing list
> > vfio-users at redhat.com
> > https://www.redhat.com/mailman/listinfo/vfio-users
> >