[vfio-users] Trouble passing PCI device in isolated IOMMU group

[Alex Williamson]

On Mon, 6 Aug 2018 17:04:24 -0600
Alex Williamson <alex.williamson at redhat.com> wrote:

> On Mon, 6 Aug 2018 16:39:51 -0600
> Quincy Wofford <quincy.wofford at gmail.com> wrote:
> 
> > I'm attempting PCI-passthrough from host to guest on an HP ProLiant 380P,
> > which has an updated, yet rather old BIOS (2015), but it does support VT-d.
> > I'm running CentOS 7, kernel 3.10.0-862.9.1.el7.x86_64.
> > 
> > I have an Intel 82580 NIC installed with 4 ports. Each of these ports is in
> > its own IOMMU group (I enabled SR-IOV at the BIOS, which might be the
> > reason they show up separately)
> > 
> >  After detaching and adding a 'hostdev' device with the appropriate pci
> > address, I attempt to start my VM. I get " failed to set iommu for
> > container: Operation not permitted". As recommended here (
> > http://vfio.blogspot.com/2014/08/vfiovga-faq.html) I parsed dmesg in an
> > attempt to find:
> > 
> > -------------------
> > No interrupt remapping support. Use the module param
> > "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform
> > -------------------
> > ...but nothing similar exists in my logs.
> > 
> > Since this device is showing up in its own IOMMU group, I assume ACS
> > override won't get me any further. In any case, it is not an option for me
> > to leave ACS override on. I can turn it on for testing, the server is not
> > currently in production.
> > 
> > Below you will find links to relevant configs:
> > 
> > virsh edit monitor: https://pastebin.com/kzCR0E5t
> > nodedev-list --tree: https://pastebin.com/AfYJKZX9
> > nodedev-dumpxml pci_0000_04_00_1: https://pastebin.com/VUurQ05Y
> > virsh start monitor: https://pastebin.com/6ANCL5pJ  
> 
> dmesg likely has the answer, probably reports:
> 
> "Device is ineligible for IOMMU domain attach due to platform RMRR
> requirement.  Contact your platform vendor."
> 
> You can read more about it here:
> 
> https://access.redhat.com/sites/default/files/attachments/rmrr-wp1.pdf
> 
> The quick summary is that the hardware vendor has imposed a mapping
> requirement for the device that is incompatible with device
> assignment.  AIUI HP for disabling these mappings for some systems,
> hopefully yours is one.  Good luck,

Oops, missed half the thought there.  Depending on the generation of
your Proliant, HP may have instructions for disabling the interfering
reserved memory ranges.  Thanks,

Alex