[vfio-users] Trouble passing PCI device in isolated IOMMU group

[Quincy Wofford]

Hi Alex,

Yes, my dmesg did report this issue related to RMRR.

Based on your suggestion, I ran the hp-script-utils steps mentioned at
https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-c04781229. I
disabled RMRR for all 3 of my available PCI slots.

I swapped NIC's in and out of different PCI slots until I found one magical
slot which just works. I inserted my 4-port Intel 82580 NIC (branded HP
365T), and then successfully started a VM on one of the 4 available ports.
None of the NIC's I tried would work in the first two slots I tried.

On Mon, Aug 6, 2018 at 5:09 PM Alex Williamson <alex.williamson at redhat.com>
wrote:

> On Mon, 6 Aug 2018 17:04:24 -0600
> Alex Williamson <alex.williamson at redhat.com> wrote:
>
> > On Mon, 6 Aug 2018 16:39:51 -0600
> > Quincy Wofford <quincy.wofford at gmail.com> wrote:
> >
> > > I'm attempting PCI-passthrough from host to guest on an HP ProLiant
> 380P,
> > > which has an updated, yet rather old BIOS (2015), but it does support
> VT-d.
> > > I'm running CentOS 7, kernel 3.10.0-862.9.1.el7.x86_64.
> > >
> > > I have an Intel 82580 NIC installed with 4 ports. Each of these ports
> is in
> > > its own IOMMU group (I enabled SR-IOV at the BIOS, which might be the
> > > reason they show up separately)
> > >
> > >  After detaching and adding a 'hostdev' device with the appropriate pci
> > > address, I attempt to start my VM. I get " failed to set iommu for
> > > container: Operation not permitted". As recommended here (
> > > http://vfio.blogspot.com/2014/08/vfiovga-faq.html) I parsed dmesg in
> an
> > > attempt to find:
> > >
> > > -------------------
> > > No interrupt remapping support. Use the module param
> > > "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform
> > > -------------------
> > > ...but nothing similar exists in my logs.
> > >
> > > Since this device is showing up in its own IOMMU group, I assume ACS
> > > override won't get me any further. In any case, it is not an option
> for me
> > > to leave ACS override on. I can turn it on for testing, the server is
> not
> > > currently in production.
> > >
> > > Below you will find links to relevant configs:
> > >
> > > virsh edit monitor: https://pastebin.com/kzCR0E5t
> > > nodedev-list --tree: https://pastebin.com/AfYJKZX9
> > > nodedev-dumpxml pci_0000_04_00_1: https://pastebin.com/VUurQ05Y
> > > virsh start monitor: https://pastebin.com/6ANCL5pJ
> >
> > dmesg likely has the answer, probably reports:
> >
> > "Device is ineligible for IOMMU domain attach due to platform RMRR
> > requirement.  Contact your platform vendor."
> >
> > You can read more about it here:
> >
> > https://access.redhat.com/sites/default/files/attachments/rmrr-wp1.pdf
> >
> > The quick summary is that the hardware vendor has imposed a mapping
> > requirement for the device that is incompatible with device
> > assignment.  AIUI HP for disabling these mappings for some systems,
> > hopefully yours is one.  Good luck,
>
> Oops, missed half the thought there.  Depending on the generation of
> your Proliant, HP may have instructions for disabling the interfering
> reserved memory ranges.  Thanks,
>
> Alex
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20180808/051ca381/attachment.htm>