[vfio-users] vfio dma map succeeds but dmesg indicates error

[Alex Williamson]

On Thu, 30 Mar 2017 13:50:27 +0000
Christopher Thompson <Chris.Thompson1 at synopsys.com> wrote:

> Hi,
> 
> I'm currently using VFIO for PCI passthrough in a custom simulation, and have discovered that when I try to map IO virtual addresses larger than my host supports, VFIO happily returns 0 from the dma map ioctl, but dmesg is telling me something is wrong.
> 
> I noticed this after a few days of wondering why this specific simulation wasn't working - the simulated system is trying to setup dma addresses using 40-bits of address space, starting at 0xfffffff000 and descending, it turns out my host only support 39 bits.
> 
> For example a dma map of 0xffffffd000, size 0x1000, the ioctl returns 0 (but if I inspect errno it has the value "Bad address"). Dmesg generates the error:
> [677414.285778] intel_iommu_map: iommu width (39) is not sufficient for the mapped address (ffffffe000)
> 
> I can accept that I can't map addresses larger than my host IOMMU supports, but shouldn't VFIO report an error from the mapping?
> 
> If this has been fixed in a later version please let me know, my Google-fu wasn't good enough to find much on this error. My kernel is a relatively ancient 3.13 (Ubuntu 14.04.4).

The kernel version may indeed be the issue, I just modified one of my
test programs to probe the address width using this function:

int probe_iova_limit(int fd, unsigned long vaddr, unsigned long pagesize)
{
        struct vfio_iommu_type1_dma_map dma_map = {
                .argsz = sizeof(dma_map),
                .flags = VFIO_DMA_MAP_FLAG_READ | VFIO_DMA_MAP_FLAG_WRITE,
                .size = pagesize,
                .vaddr = vaddr,
        };
        struct vfio_iommu_type1_dma_unmap dma_unmap = {
                .argsz = sizeof(dma_unmap),
                .size = pagesize,
        };
        int ret;
        unsigned long iova = pagesize;

        while (iova) {
                dma_map.iova = dma_unmap.iova = iova - pagesize;

                ret = ioctl(fd, VFIO_IOMMU_MAP_DMA, &dma_map);
                if (ret) {
                        printf("IOMMU failed at 0x%lx (%m), address width %d\n",
                               dma_map.iova, ffsll(iova) - 2);
                        return 0;
                }

                ret = ioctl(fd, VFIO_IOMMU_UNMAP_DMA, &dma_unmap);
                if (ret) {
                        printf("Unmap 0x%lx failed\n", dma_unmap.iova);
                        return ret;
                }

                iova <<= 1;
        }

        printf("IOMMU has full 64 bit address width, no failures\n");
        return 0;
}

And my BDW Intel NUC reports:

IOMMU failed at 0xfffffff000 (Bad address), address width 39

An X79 box reports:

IOMMU failed at 0x1fffffffff000 (Bad address), address width 48

Tested both a v4.10 kernel and RHEL7 kernel on the NUC, RHEL7 on the
X79.  Try a different kernel, let me know what you find.  Thanks,

Alex