[libvirt] NFS over RDMA small block DIRECT_IO bug

[Steve Dickson]

On 09/04/2012 05:31 AM, Andrew Holway wrote:
> Hello.
> 
> # Avi Kivity avi(a)redhat recommended I copy kvm in on this. It would also seem relevent to libvirt. #
> 
> I have a Centos 6.2 server and Centos 6.2 client.
> 
> [root at store ~]# cat /etc/exports 
> /dev/shm				10.149.0.0/16(rw,fsid=1,no_root_squash,insecure)    (I have tried with non tempfs targets also)
> 
> 
> [root at node001 ~]# cat /etc/fstab 
> store.ibnet:/dev/shm             /mnt                 nfs          rdma,port=2050,defaults 0 0
> 
> 
> I wrote a little for loop one liner that dd'd the centos net install image to a file called 'hello' then checksummed that file. Each iteration uses a different block size.
> 
> Non DIRECT_IO seems to work fine. DIRECT_IO with 512byte, 1K and 2K block sizes get corrupted.
> 
> I want to run my KVM guests on top of NFS over RDMA. My guests cannot create filesystems.
> 
> Thanks,
> 
> Andrew.
> 
> bug report: https://bugzilla.linux-nfs.org/show_bug.cgi?id=228
Well it appears the RHEL6 kernels are lacking a couple patches that might
help with this....

5c635e09 RPCRDMA: Fix FRMR registration/invalidate handling.
9b78145c xprtrdma: Remove assumption that each segment is <= PAGE_SIZE

I can only image that Centos 6.2 might me lacking these too... ;-)

steved.