[libvirt] Exposing mem-path in domain XML

[Zack Cornelius]

----- Original Message -----
> From: "Michal Privoznik" <mprivozn at redhat.com>
> To: "Zack Cornelius" <zack.cornelius at kove.net>
> Cc: "libvir-list" <libvir-list at redhat.com>
> Sent: Friday, September 29, 2017 2:44:13 AM
> Subject: Re: [libvirt] Exposing mem-path in domain XML


> For this configuration, two memory-backend-file objects are created. The
> first one is for the guest RAM (for the NUMA node), second is for the
> DIMM module. While the alias for the DIMM module is exposed in the XML,
> the alias for the NUMA node is missing:
> 
> -object
> memory-backend-file,id=ram-node0,mem-path=/var/lib/libvirt/qemu/ram,share=yes,size=4294967296
> -numa node,nodeid=0,cpus=0-3,memdev=ram-node0
> -object
> memory-backend-file,id=memdimm0,mem-path=/var/lib/libvirt/qemu/ram,share=yes,size=536870912
> -device pc-dimm,node=0,memdev=memdimm0,id=dimm0,slot=0
> 
> This complicates things IMO. I guess what I'm saying is that we can
> generate full paths including the device alias as filename, but I'm not
> sure it is going to help, is it?
> 

Looking at the examples, it seems like these automatically added memory-backend-file objects would always be generated in a consistent manner. As long as this is true, and these objects always end up with a consistent filenames, our integration components can assume the filenames in a consistent way.

For instance, as long as every numa <cell> element which has memory defined generates a file named ram-node<numa_node_number>, we can handle that mapping within our integration components by parsing the XML and identifying the NUMA nodes with memory, and allocating an appropriate amount, with the correct filename.

> Alternatively, and I admit I don't know much about Kove, if libvirt
> would put all the files into per-domain directories, would that be
> enough for you? In the example above all the files are put into generic
> path. However, if the path looked like this:
> 
> $memoryBackingDir/$domain/
> 
> You could differentiate which files belong to which domain.
> 

For our integration to work well for users, we need to allocate the external memory prior to qemu starting, so we can give descriptive error messages as part of our integration components. 

The directory based approach would require us to allocate external memory when the individual files are created or mmap'd in qemu. This may lead to a situation where there is insufficient external memory, leading to the failure happening within qemu, and a much more generic qemu unable to allocate memory error.

With the external memory allocations and files being created and sized prior to qemu startup via our integration components called via the libvirt prepare hook, we can detect this situation during the libvirt hook, and return a more appropriate error message.


--Zack