[libvirt][PATCH v5 0/3] introduce 'restrictive' mode in numatune

[Martin Kletzander]

On Tue, Apr 13, 2021 at 02:38:05PM +0800, Luyao Zhong wrote:
>Before this patch set, numatune only has three memory modes:
>static, interleave and prefered. These memory policies are
>ultimately set by mbind() system call.
>
>Memory policy could be 'hard coded' into the kernel, but none of
>above policies fit our requirment under this case. mbind() support
>default memory policy, but it requires a NULL nodemask. So obviously
>setting allowed memory nodes is cgroups' mission under this case.
>So we introduce a new option for mode in numatune named 'restrictive'.
>
><numatune>
>   <memory mode="restrictive" nodeset="1-4,^3"/>
>   <memnode cellid="0" mode="restrictive" nodeset="1"/>
>   <memnode cellid="2" mode="restrictive" nodeset="2"/>
></numatune>
>
>The config above means we only use cgroups to restrict the allowed
>memory nodes and not setting any specific memory policies explicitly.
>
>For this new "restrictive" mode, there is a concrete use case about a
>new feature in kernel but not merged yet, we call it memory tiering.
>(https://lwn.net/Articles/802544/).
>If memory tiering is enabled on host, DRAM is top tier memory, and
>PMEM(persistent memory) is second tier memory, PMEM is shown as numa node
>without cpu. Pages can be migrated between DRAM node and PMEM node based on
>DRAM pressure and how cold/hot they are. *this memory policy* is implemented
>in kernel. So we need a default mode here, but from libvirt's perspective,
>the "defaut" mode is "strict", it's not MPOL_DEFAULT
>(https://man7.org/linux/man-pages/man2/mbind.2.html) defined in kernel.
>And to make memory tiering works well, cgroups setting is necessary, since
>it restricts that the pages can only be migrated between the DRAM and PMEM
>nodes that we specified (NUMA affinity support).
>
>Just using cgroups with multiple nodes in the nodeset makes kernel decide
>on which node (out of those in the restricted set) to allocate on, but specifying
>"strict" basically allocates it sequentially (on the first one until it is full,
>then on the next one and so on).
>
>In a word, if a user requires default mode(MPOL_DEFAULT), that means they want
>kernel decide the memory allocation and also want the cgroups to restrict memory
>nodes, "restrictive" mode will be useful.
>

I applied the changes locally and fixed some changes that happened in
the meantime.  I also split the patches differently as we usually add
conf, docs and schemas (driver-unrelated code) and some possible tests
in one patch and then add support for each applicable driver in separate
patches.  I reworded some comments there were also two memory leaks that
I fixed and I will resend the series later to see if we have everything
in order.

If we disagree on the naming, then we can change it until the release,
but I do not think that is something that should stall the patches.

Thanks.

>BR,
>Luyao
>
>Luyao Zhong (3):
>  docs: add docs for 'restrictive' option for mode in numatune
>  schema: add 'restrictive' config option for mode in numatune
>  qemu: add parser and formatter for 'restrictive' mode in numatune
>
> docs/formatdomain.rst                         |  7 +++-
> docs/schemas/domaincommon.rng                 |  2 +
> include/libvirt/libvirt-domain.h              |  1 +
> src/conf/numa_conf.c                          |  9 ++++
> src/qemu/qemu_command.c                       |  6 ++-
> src/qemu/qemu_process.c                       | 27 ++++++++++++
> src/util/virnuma.c                            |  3 ++
> .../numatune-memnode-invalid-mode.err         |  1 +
> .../numatune-memnode-invalid-mode.xml         | 33 +++++++++++++++
> ...emnode-restrictive-mode.x86_64-latest.args | 38 +++++++++++++++++
> .../numatune-memnode-restrictive-mode.xml     | 33 +++++++++++++++
> tests/qemuxml2argvtest.c                      |  2 +
> ...memnode-restrictive-mode.x86_64-latest.xml | 41 +++++++++++++++++++
> tests/qemuxml2xmltest.c                       |  1 +
> 14 files changed, 201 insertions(+), 3 deletions(-)
> create mode 100644 tests/qemuxml2argvdata/numatune-memnode-invalid-mode.err
> create mode 100644 tests/qemuxml2argvdata/numatune-memnode-invalid-mode.xml
> create mode 100644 tests/qemuxml2argvdata/numatune-memnode-restrictive-mode.x86_64-latest.args
> create mode 100644 tests/qemuxml2argvdata/numatune-memnode-restrictive-mode.xml
> create mode 100644 tests/qemuxml2xmloutdata/numatune-memnode-restrictive-mode.x86_64-latest.xml
>
>-- 
>2.25.4
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/libvir-list/attachments/20210416/00302ef6/attachment-0001.sig>