[libvirt] adding a new libvirt xml element for File Descriptor backed memory for use with vhost-user

[Mooney, Sean K]

Hi Daniel
Thanks for your response.
Comments inline.
Regards
Sean.

> -----Original Message-----
> From: Daniel P. Berrange [mailto:berrange at redhat.com]
> Sent: Wednesday, May 11, 2016 10:44 AM
> To: Mooney, Sean K <sean.k.mooney at intel.com>
> Cc: libvir-list at redhat.com
> Subject: Re: [libvirt] adding a new libvirt xml element for File
> Descriptor backed memory for use with vhost-user
> 
> On Tue, May 10, 2016 at 09:01:20PM +0000, Mooney, Sean K wrote:
> > Today the only native  way to request Libvirt to create a vm who's
> > memory is backed by a  memdev which can be Accessed via a file
> > descriptor is to request hugepage backed memory. This requires the
> > operator to Manage and configure hugepage on each of their compute
> hosts and take special care to ensure that vms are Not placed on host
> were vhost-user interface are used if they do not request hugepages.
> >
> > Today it is possible to use Libvirt to spawn a vm without hugepage
> > memory and a file descriptor backed memdev Via the use of the
> qemu:commandline element.
> >
> >   <qemu:commandline>
> >     <qemu:arg value='-object'/>
> >     <qemu:arg value='memory-backend-file,id=mem,size=1024M,mem-
> path=/var/lib/libvirt/qemu,share=on'/>
> >     <qemu:arg value='-numa'/>
> >     <qemu:arg value='node,memdev=mem'/>
> >     <qemu:arg value='-mem-prealloc'/>
> >   </qemu:commandline>
> >
> > I created a proof of concept patch to nova to demonstrate that this
> > works however to support this usecase in Nova a new xml element is
> required.
> > https://review.openstack.org/#/c/309565/1
> >
> > I would like to propose the introduction of  a new subelemnt to the
> > memorybacking element to request file discrptro backed memory
> >
> > <memoryBacking>
> >    <filedescriptor size_mb="1024" path="/var/lib/libvirt/qemu"
> > prealloc="true" shared="on" />  </memoryBacking>
> 
> Specifying a size is not required - we already know how big memory must
> be for the guest.
> 
> We already have a memAccess='shared' attribute against the <numa>
> element that is used to determine if the underlying memory should be
> setup as shared.  We could define a further element that lets us control
> memory access mode for guests without NUMA topology specified.
[Mooney, Sean K] hi yes the reason I added the shared attribute was to cater for
The case of guest without numa topology. For guest with numa topology I agree that
Using the memAcess='shared' on the cell is better for consistency with hugepage memory.
> 
>   <memoryBacking>
>      <access mode="shared"/>
>   </memoryBacking>
> 
> For huge pages it seems we unconditionally pass --mem-prealloc. I'm
> thinking we could perhaps make that configurable via an element
> 
> 
>   <memoryBacking>
>      <allocation mode="immediate|ondemand"/>
>   </memoryBacking>
> 
> to control use of -mem-prealloc or not.
[Mooney, Sean K] for the vhost user case the the mem-prealloc is required
Because you are basically doing dma so you really want memory to allocated.
Generically though from a Libvirt point of view I do think It makes sense for this
To be configurable to allow over subscript of memory for higher density.
> 
> So all that remains is a way to request file based backing of RAM. As
> with huge pages, I think we should hide the actual path from the user.
> We should just use /dev/shm as the backing for non-hugepage RAM. For
> this we could define something like
> 
>    <memoryBacking>
>        <source type="file|anonymous"/>
>    </memoryBacking>
> 
[Mooney, Sean K] for some reason when I used /dev/shm I could only boot one instance at a time.
that was my first choice but maybe we would have to create a file per instance under /dev/shm to make it work.
> 
> Putting that all together, to get what you want we'd have
> 
>    <memoryBacking>
>        <source type="file"/>
>        <access mode="shared"/>
>        <allocation mode="immediate"/>
>    </memoryBacking>
> 
[Mooney, Sean K] 
Yes this seems like it would be a clean way to address this use case.
Can you guage how small/large of a change this would be. Its been
A while since I worked with c directly but if you could point me in the
Right direction in the Libvirt  codebase I would be happy to look at
creating an RFC patch. 

>From a nova side assuming Libvirt was extended for this feature should
I open a blueprint to extend the existing guest memory backing support
In parallel to the Libvirt implementation or wait until after it is 
support in Libvirt to start the Nova discussion? In either case I think
we agree that any support in nova Would Depend on Libvirt support to be
accepted in  upstream nova.

> 
> Regards,
> Daniel
> --
> |: http://berrange.com      -o-
> http://www.flickr.com/photos/dberrange/ :|
> |: http://libvirt.org              -o-             http://virt-
> manager.org :|
> |: http://autobuild.org       -o-
> http://search.cpan.org/~danberr/ :|
> |: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-
> vnc :|