[vfio-users] Question about DMA from user space buffers

[Alex Williamson]

On Fri, 30 Nov 2018 23:56:34 +0000
Nikos Dragazis <ndragazis at outlook.com.gr> wrote:

> Hi all,
> 
> I am trying to understand how vfio enables DMA from user space memory. I
> would appreciate any help on this.
> 
> Suppose we have a system with x86 architecture and an IOMMU. When a user
> space driver in that system wants to ask from a PCI device to perform
> DMA from user space memory, it has to register that memory with
> VFIO_IOMMU_MAP_DMA ioctl. What are the restrictions that this user space
> memory has to meet? How are the DMA memory limitations met? Is this
> totally arranged by vfio? Or is it a responsibility of the user space
> process (eg. use hugepages)?
> 
> I had a conversation on the SPDK mailing list about this. SPDK is a
> storage framework running entirely in user space and uses vfio to
> control the PCI storage devices. This is the relevant email:
> https://lists.01.org/pipermail/spdk/2018-November/002733.html
> 
> I am a little bit confused about the purpose of hugepages in SPDK and
> whether they are related to vfio.
> 
> To summarize, I am trying to find the answer to the following questions:
> 
> Is there any constraint about the DMA memory buffers passed to vfio?

They need to match the minimum page size and alignment requirements of
the IOMMU, ex. 4K on x86.

The user specified IOVA needs to fit within the physical limits of the
IOMMU, typically 39 or 48 bits on Intel systems.

> Is it possible to pass a buffer allocated with malloc?

Yep.

> Or is it necessary to use hugepages which by design serve all the DMA
> memory limitations (pinned, non-swappable, physically contiguous)?

Not required, vfio will pin pages itself, physically discontiguous is
handled transparently in the mapping ioctl.  Using hugepages will see
benefits that we'll forward those contiguous mappings to the IOMMU,
which can then use superpages in the IOMMU page tables.

Regarding your "DMA memory limitations" question above, I'm not sure if
this is referring to limited DMA address space of the device, but the
user specifies the IOVA, so it's up to them to specify an address
reachable by the device if such a limit exists.  Thanks,

Alex