[vfio-users] Question about DMA from user space buffers

[Nikos Dragazis]

On 1/12/18 2:23 π.μ., Alex Williamson wrote:

> On Fri, 30 Nov 2018 23:56:34 +0000
> Nikos Dragazis <ndragazis at outlook.com.gr> wrote:
>
>> Hi all,
>>
>> I am trying to understand how vfio enables DMA from user space memory. I
>> would appreciate any help on this.
>>
>> Suppose we have a system with x86 architecture and an IOMMU. When a user
>> space driver in that system wants to ask from a PCI device to perform
>> DMA from user space memory, it has to register that memory with
>> VFIO_IOMMU_MAP_DMA ioctl. What are the restrictions that this user space
>> memory has to meet? How are the DMA memory limitations met? Is this
>> totally arranged by vfio? Or is it a responsibility of the user space
>> process (eg. use hugepages)?
>>
>> I had a conversation on the SPDK mailing list about this. SPDK is a
>> storage framework running entirely in user space and uses vfio to
>> control the PCI storage devices. This is the relevant email:
>> https://lists.01.org/pipermail/spdk/2018-November/002733.html
>>
>> I am a little bit confused about the purpose of hugepages in SPDK and
>> whether they are related to vfio.
>>
>> To summarize, I am trying to find the answer to the following questions:
>>
>> Is there any constraint about the DMA memory buffers passed to vfio?
> They need to match the minimum page size and alignment requirements of
> the IOMMU, ex. 4K on x86.
>
> The user specified IOVA needs to fit within the physical limits of the
> IOMMU, typically 39 or 48 bits on Intel systems.
>
>> Is it possible to pass a buffer allocated with malloc?
> Yep.
>
>> Or is it necessary to use hugepages which by design serve all the DMA
>> memory limitations (pinned, non-swappable, physically contiguous)?
> Not required, vfio will pin pages itself, physically discontiguous is
> handled transparently in the mapping ioctl.  Using hugepages will see
> benefits that we'll forward those contiguous mappings to the IOMMU,
> which can then use superpages in the IOMMU page tables.
>
> Regarding your "DMA memory limitations" question above, I'm not sure if
> this is referring to limited DMA address space of the device, but the
> user specifies the IOVA, so it's up to them to specify an address
> reachable by the device if such a limit exists.  Thanks,
>
> Alex

Thanks a lot for your really quick reply.

No doubt that the IOVA address width is a limitation. What I actually
had in mind when I was referring to "DMA memory limitations" was the
following constraints:

- user space buffers have to be mapped to physical memory frames (some
  pages may haven't been mapped to physical memory frames yet due to
  demand paging or some pages could have been swapped out).
- user space buffers have to be pinned in memory. This means that they
  will never get swapped-out and the VA-to-PA translation will be fixed.
- user space buffers that are possibly scattered in physical memory have
  to be exposed to the device as physically contiguous. This implies
  either that the entire buffer has to be physically contiguous (eg.
  hugepages or small enough to fit in 4KB) or that the scattered
  physical memory frames making up the buffer have to be mapped to a
  single contiguous IOVA segment.

Based on your answer, I understand that all these are satisfied by vfio.