[vfio-users] vfio fails Guest FreeBSD9.3 host Fedora 23

[Bob Dawes]

On 27/05/16 20:11, Alex Williamson wrote:
> [re-adding vfio-users]
>
> On Fri, 27 May 2016 15:03:17 -0400
> chintu hetam <rometoroam at gmail.com> wrote:
>
>> that worked Alex, sorry for the delay in response.
>> so my memorybacking configuration was
>>   <memoryBacking>
>>      <hugepages>
>>        <page size='1048576' unit='KiB' nodeset='1'/>
>>      </hugepages>
>>      <nosharepages/>
>>      <locked/>
>>    </memoryBacking>
>> i had to remove it completely in order to make it work.
>>
>> So confirming that hugepages and vfio has some issue, making VM to hang in
>> case of linux at startup and abruptly crashing from FreeBSD guest during
>> startup.
> Can you see if the problem persists with updated QEMU from virt-preview:
>
> https://fedoraproject.org/wiki/Virtualization_Preview_Repository
>
> 2MB hugepages certainly works, I run it daily.  1GB hugepages sees less
> attention, but I certainly thought it worked as well.  Thanks,
>
> Alex
>
> _______________________________________________
> vfio-users mailing list
> vfio-users at redhat.com
> https://www.redhat.com/mailman/listinfo/vfio-users

I use both 2Mb and 1Gb pages in a pair of numa nodes and have never had 
any problems with them except when memory has fragmented before I launch 
the emulator. I've done this with qemu command line since about 2.4, but 
I do remember 2Mb and 1Gb pages weren't supported on the same machine 
until around kernel 2.4 ... or maybe they required a lot of work. Maybe 
freebsd is like that.

Have you checked your hugepages directory? it needs to be writable to 
the process running qemu and large enough to make the given files. 
Libvirt may need something I don't know about especially if running as a 
normal user.

I tend to have more than one mount point that has the fixed size I 
require and the correct permissions / gid so the parameters are all in 
one place and it's easy to test.

echo 3584 > "/sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages"

mount -t hugetlbfs -o 
uid=0,gid=124,mode=0777,size=7168M,nr_inodes=3584,pagesize=2M hugetlbfs 
$CDIR/qmp/hugepagesfs

There's a lot of ways to break hugepages in the kernel /proc/sys /sysfs 
as well. I think if you don't use pre-alloc in qemu the pages can get 
claimed and freed and fragmented / swapped out.

One last thought ... you don't need the balloon device if you've got a 
fixed pre-allocated memory size as below

-m 32768 -mem-prealloc -mem-path /dev/hugepages/libvirt/qemu

doubt it matters ... although I also have trouble with some passthrough 
devices if I don't use -cpu host with kvm and I have no idea why.