[vfio-users] vfio_pci_mmap failure : Hitting mmaps overlapping the MSI-X table

[Vikas Aggarwal]

Hello list vfio-users,
Can someone help me understand reason that why mmap of requested address
overlaps with MSI-X table during mmap-ing of PCIe resources.

Platform      :  ARM64 architecture (Marvell OcteonTX2)

Linux kernel:  4.14.76-22.0.0 aarch64,  Page Size 64K

Application :  Userspace DPDK+SPDK doing  mmap-ing of PCIe resources  via
pci_vfio_map_resource_primary( )

http://code.dpdk.org/dpdk/v19.11/source/lib/librte_pci/rte_pci.c#L140
                          mapaddr  = mmap(0x202080040000, 0x3000,
PROT_READ | PROT_WRITE, MAP_SHARED, 35, 0x0);

Device 0003:0d:00.0  :   Samsung SSD

Failure:         mapaddr  returned is all 0xffffffffffffffff, errno is set
to EINVAL
                     EAL: pci_map_resource(): cannot mmap(36,
0x2020801e0000, 0x2000, 0x0): Invalid argument (0xffffffffffffffff)
	       EAL: Failed to map pci BAR0
                     EAL:   0003:0d:00.0 mapping BAR0 failed: Invalid
argument
                     EAL: Requested device 0003:0d:00.0 cannot be used

Cause from kernel mmap handler:
EINVAL is returned by vfio_pci_mmap()  in-kernel handler :
https://elixir.bootlin.com/linux/v4.14.76/source/drivers/vfio/pci/vfio_pci
.c#L1142
	if (index == vdev->msix_bar) {
		/*
		 * Disallow mmaps overlapping the MSI-X table; users don't
		 * get to touch this directly.  We could find somewhere
		 * else to map the overlap, but page granularity is only
		 * a recommendation, not a requirement, so the user needs
		 * to know which bits are real.  Requiring them to mmap
		 * around the table makes that clear.
		 */

		/* If neither entirely above nor below, then it overlaps
*/
		if (!(req_start >= vdev->msix_offset + vdev->msix_size ||
		      req_start + req_len <= vdev->msix_offset))
			return -EINVAL;  <=====================Hitting
this
	}
>From Debug prints:
              req_start = 0; vdev->msix_offset = 8192;
vdev->msix_size=144;  req_len=65536,  vdev->msix_offset=8192;

Can someone explain me how come this overlap situation is coming and how
can I fix it.
Thanks very much
Vikas Aggarwal
Sr. Principal Engineer
Diamanti Inc
www.diamanti.com