[vfio-users] vfio dma map succeeds but dmesg indicates error

[Christopher Thompson]

Hi Alex,

Thanks very much, I'll lean on IT a bit to try and test this with a newer kernel, but it looks like it's definitely been addressed in later kernels. What I really need is to get a new system with wider IOVA support.

Obviously your X79 board supports 48-bits, looking through Intel's datasheets it looks like the base level VT-d supports 39-bits, while some CPUs support 48-bits (the 4xxx 5xxx 6xxx and 7xxx series at least, but not 3xxx or older):
"The processor supports the following added new Intel VT-d features:
 - 4-level Intel VT-d Pagewalk - bothe default Intel VT-d engine as well as the IGD VT-d engine are upgraded to support 4-level Intel VT-d tables (adjusted guest address width of 48 bits)"

Do you know if the chipset would put any limit on this (so long as the chipset supports VT-d in the first place)?

For full disclosure I'm the same Chris Thompson as emailed the list a while ago from the_cartographer at hotmail.com, I initially used a personal email because I prefer to not clutter up my work inbox with mailing lists, but recent policy changes at work mean I can't actually access personal email from my work machine any more.

Thanks and regards,
Chris

> -----Original Message-----
> From: Alex Williamson [mailto:alex.williamson at redhat.com]
> Sent: 30 March 2017 17:18
> To: Christopher Thompson <Chris.Thompson1 at synopsys.com>
> Cc: vfio-users at redhat.com
> Subject: Re: [vfio-users] vfio dma map succeeds but dmesg indicates error
> 
> On Thu, 30 Mar 2017 13:50:27 +0000
> Christopher Thompson <Chris.Thompson1 at synopsys.com> wrote:
> 
> > Hi,
> >
> > I'm currently using VFIO for PCI passthrough in a custom simulation, and
> have discovered that when I try to map IO virtual addresses larger than my
> host supports, VFIO happily returns 0 from the dma map ioctl, but dmesg is
> telling me something is wrong.
> >
> > I noticed this after a few days of wondering why this specific simulation
> wasn't working - the simulated system is trying to setup dma addresses using
> 40-bits of address space, starting at 0xfffffff000 and descending, it turns out
> my host only support 39 bits.
> >
> > For example a dma map of 0xffffffd000, size 0x1000, the ioctl returns 0 (but
> if I inspect errno it has the value "Bad address"). Dmesg generates the error:
> > [677414.285778] intel_iommu_map: iommu width (39) is not sufficient
> > for the mapped address (ffffffe000)
> >
> > I can accept that I can't map addresses larger than my host IOMMU
> supports, but shouldn't VFIO report an error from the mapping?
> >
> > If this has been fixed in a later version please let me know, my Google-fu
> wasn't good enough to find much on this error. My kernel is a relatively
> ancient 3.13 (Ubuntu 14.04.4).
> 
> The kernel version may indeed be the issue, I just modified one of my test
> programs to probe the address width using this function:
> 
> int probe_iova_limit(int fd, unsigned long vaddr, unsigned long pagesize) {
>         struct vfio_iommu_type1_dma_map dma_map = {
>                 .argsz = sizeof(dma_map),
>                 .flags = VFIO_DMA_MAP_FLAG_READ |
> VFIO_DMA_MAP_FLAG_WRITE,
>                 .size = pagesize,
>                 .vaddr = vaddr,
>         };
>         struct vfio_iommu_type1_dma_unmap dma_unmap = {
>                 .argsz = sizeof(dma_unmap),
>                 .size = pagesize,
>         };
>         int ret;
>         unsigned long iova = pagesize;
> 
>         while (iova) {
>                 dma_map.iova = dma_unmap.iova = iova - pagesize;
> 
>                 ret = ioctl(fd, VFIO_IOMMU_MAP_DMA, &dma_map);
>                 if (ret) {
>                         printf("IOMMU failed at 0x%lx (%m), address width %d\n",
>                                dma_map.iova, ffsll(iova) - 2);
>                         return 0;
>                 }
> 
>                 ret = ioctl(fd, VFIO_IOMMU_UNMAP_DMA, &dma_unmap);
>                 if (ret) {
>                         printf("Unmap 0x%lx failed\n", dma_unmap.iova);
>                         return ret;
>                 }
> 
>                 iova <<= 1;
>         }
> 
>         printf("IOMMU has full 64 bit address width, no failures\n");
>         return 0;
> }
> 
> And my BDW Intel NUC reports:
> 
> IOMMU failed at 0xfffffff000 (Bad address), address width 39
> 
> An X79 box reports:
> 
> IOMMU failed at 0x1fffffffff000 (Bad address), address width 48
> 
> Tested both a v4.10 kernel and RHEL7 kernel on the NUC, RHEL7 on the X79.
> Try a different kernel, let me know what you find.  Thanks,
> 
> Alex