[libvirt] [RFC] libvirt hugepage hooks

[Alex Williamson]

Hi,

This is very rough and early, but I wanted to get some feedback,
possibly advice, and see if there's some interest in at least creating
infrastructure for user contributed libvirt hooks, if not some default
ones that are no-ops unless configured.

The impetus for this is that I started trying to tune a dual-socket
system for performance with device assignment and quickly became very
frustrated that distros don't provide any built-in support for more than
the very basics of hugepages.  Yes, there are kernel commandline
options, but those don't allow node specific configuration.  Yes,
there's 'virsh allocpages', but how does that get incorporated
automatically into initializing libvirtd or a domain?  Creating any sort
of persistence for hugepages is an exercise for the user.

So I think the first step in this is that the hooks scripts[1] should by
default support sub-scripts in the common way, with a ".d" sub-directory
holding those scripts, for example daemon.d and qemu.d.  In the attached
file, I've simply commandeered the default script to call the
sub-scripts.  For compatibility (ie. not overwriting user scripts), that
should probably happen within libvirt.  In any case, a single monolithic
hook file is impossible to maintain on a system, let alone multiple
systems, so this needs to be brought up to date.

The second step is that even if we drop user contrib hooks out
in /usr/share for admins to pull in as desired, perhaps we can provide
some consistency for how to configure those hooks.  In the example below
I propose /etc/sysconfig/libvirt-hook-config.xml.  You can see how
currently it supports static and dynamic hugepage hooks, static
occurring through the daemon hook and dynamic through the qemu hook.
Ideally the dynamic hook would simply list the domain names
participating in dynamic hugepages and figure out what needs to be
allocated where from the domain xml.  I haven't gotten that far (and
frankly trying to satisfy cpu/numa vs numatune/memory|memnode vs
memoryBacking/hugepages and memory size still looks very confusing to
me).  Do we want a common place to configure this sort of thing?  Is XML
the right format?

On to the scripts themselves.  I got some advice on #virt that I should
use 'virsh allocpages' to manage hugepages.  Despite the warning not to
call into libvirt in the hook documentation[1], I was assured it'd be ok
here.  However, somehow 'virsh freepages' did manage to hang and my
RHEL7.1 system doesn't support allocpages yet, so my prototype uses raw
sysfs.  AFAICT, any sort of hugepage manipulation is inherently broken
because of the racy kernel interfaces.  We really need a hugepage
broker, but that's well beyond the scope of libvirt.

Functionally this seems to work well for me.  I don't know how practical
it is to support dynamic 1G pages; I'd probably encourage static setup
for that as my system only survived a couple rounds before getting too
fragmented.  2M dynamic seems to work quite nicely though.

TL;DR, I thought I'd post this, even in a rough state to see if there's
interest, get nitpicks at my terrible scripting, and make sure I'm not
just scratching my own itch.  Thanks,

Alex

[1] https://www.libvirt.org/hooks.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: libvirt-hook-config.xml
Type: application/xml
Size: 607 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/libvir-list/attachments/20150616/ceb3a95c/attachment-0001.wsdl>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: daemon
Type: application/x-shellscript
Size: 192 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/libvir-list/attachments/20150616/ceb3a95c/attachment-0004.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: qemu
Type: application/x-shellscript
Size: 327 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/libvir-list/attachments/20150616/ceb3a95c/attachment-0005.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: static-hugepages.sh
Type: application/x-shellscript
Size: 2899 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/libvir-list/attachments/20150616/ceb3a95c/attachment-0006.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: dynamic-hugepages.sh
Type: application/x-shellscript
Size: 3096 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/libvir-list/attachments/20150616/ceb3a95c/attachment-0007.bin>