[vfio-users] Application examples using vfio ?

[Ran Shalit]

On Tue, Feb 6, 2018 at 5:41 PM, Alex Williamson
<alex.williamson at redhat.com> wrote:
> On Tue, 6 Feb 2018 17:20:08 +0200
> Ran Shalit <ranshalit at gmail.com> wrote:
>
>> On Tue, Feb 6, 2018 at 4:56 PM, Alex Williamson
>> <alex.williamson at redhat.com> wrote:
>> > On Tue, 6 Feb 2018 16:24:06 +0200
>> > Ran Shalit <ranshalit at gmail.com> wrote:
>> >
>> >> Hello,
>> >>
>> >> I need to write vfio userspace driver for pci express.
>> >> I've been searching for some examples of using vfio, but did not find any,
>> >> Is there a git/examples of userspace application using vfio ?
>> >
>> > git://git.qemu.org/qemu.git
>> > git://dpdk.org/dpdk
>> > https://github.com/andre-richter/rVFIO
>> > https://github.com/MicronSSD/unvme
>> > https://github.com/awilliam/tests
>>
>> Thank you.
>> I am trying to understand how to use it with dma.
>> I've been checking now the links, but did not find yet in the above
>> example how to both map DMA  memory and read/write from that memory.
>> I see that the above test git contains tests which map/unmap DMA, but
>> not reading/writing from that dma memory afterwards (there is a
>> seperate test which read/write from memory but without dma).
>> Is that correct or did I miss something ? Is there any example which
>> shows how to both map and then read/write ?
>
> Initiating DMA is device specific, you need to understand the
> programming model of the device for that.

I haven't yet found any example, so I please want to ask one more on dma issue:
As far as I know, in kernel drivers, you can do the following:
1. allocate dma memory
2. program the device to be familiar with the dma address,
3. and then for write:  trigger a dma transaction by writing in BARs
specific register.
4. and for read:  usually just waiting for interrupt
So, if I understand correctly in the dma test example , only (1) above
is implemented, and the rest are not implemented becuase they are
device specific (and the test are generic), Right ?

I would also like to ask about interrupt, I read the documentation in
vfio.h, but I can't find how the irq is actually set.
I don't see that VFIO_DEVICE_SET_IRQS
has parameter for irq number. Does it ?

Thank you,
Ran


 The vfio DMA ioctls only map
> the buffers through the IOMMU to allow that DMA, they don't control the
> device.  The unit tests in the last link doesn't have any device
> specific code, they're only performing the mapping without actually
> initiating device DMA.  The QEMU examples also mostly configure DMA but
> rely on native drivers within the guest VM to program the device to
> initiate DMA.  The rVFIO project is also just a wrapper to make vfio
> more accessible to scripting languages, so I don't expect actual device
> driver code there.  DPDK and the NVME driver should have actual driver
> code for the device though.  In fact there's also an NVME project
> within QEMU that makes use of vfio that might have driver code.  The
> simple logic though is that the DMA map ioctl takes a user provided
> virtual buffer and maps it through the IOMMU at the user provided IOVA
> (I/O virtual address).  It's up to the user to make the device perform
> DMA using that IOVA.  Note that the IOMMU address width is typically
> less than the user virtual address space, so identity mapping the IOVA
> to the virtual address is not a good strategy.
>
>> Another question, just to be sure I'm on the right direction. I wanted
>> to write pci express userspace driver (using real device, and not
>> related to virtualization). Is vfio a proper solution for this kind of
>> thing ?
>
> Yes, that's exactly what DPDK does.
>
>> I ask because there are no plenty of examples as I first expected, and
>> I try to understand why there are no many, (probably license issues ?)
>
> vfio is of course part of the kernel and GPL, but userspace drivers are
> not necessarily under the same license, so it is possible that the
> drivers themselves are more private.  DPDK supports several NICs and
> I'd guess the NVME driver handles the standard NVME programming model
> and could therefore support devices from many vendors.  Thanks,
>
> Alex