Memory locking limit and zero-copy migrations

[Jiri Denemark]

On Fri, Aug 19, 2022 at 08:01:58 +0200, Milan Zamazal wrote:
> Fangge Jin <fjin at redhat.com> writes:
> 
> > On Fri, Aug 19, 2022 at 4:08 AM Milan Zamazal <mzamazal at redhat.com> wrote:
> >
> >> > Not sure whether you already know this, but I had a hard time
> >> > differentiating the two concepts:
> >> > 1. memlock hard limit(shown by prlimit): the hard limit for locked host
> >> > memory
> >> > 2. memtune hard limit(memtune->hard_limit): the hard limit for in-use
> >> host
> >> > memory, this memory can be swapped out.
> >>
> >> No, I didn't know it, thank you for pointing this out.  Indeed, 2. is
> >> what both the libvirt and kernel documentation seem to say, although not
> >> so clearly.
> >>
> >> But when I add <memtune> with <hard_limit> to the domain XML and then
> >> start the VM, I can see the limit shown by `prlimit -l' is increased
> >> accordingly.  This is good for my use case, but does it match what you
> >> say about the two concepts?
> >
> > memtune->hard_limit(hard limit of in-use memory) actually takes effect via
> > cgroup,
> > you can check the value by:
> > # virsh memtune uefi1
> > hard_limit     : 134217728
> > soft_limit     : unlimited
> > swap_hard_limit: unlimited
> > # cat
> > /sys/fs/cgroup/memory/machine.slice/machine-qemu\\x2d6\\x2duefi1.scope/libvirt/memory.limit_in_bytes
> >
> > 137438953472
> >
> > When vm starts with memtune->hard_limit set in domain XML, memlock
> > hard limit( hard_limit of locked memory, shown by 'prlimit -l')will be
> > set to the value of memtune->hard_limit. This's probably because
> > memlock hard limit must be less than memtune->hard_limit.
> 
> Well, increasing the memlock limit to keep it within memtune->hard_limit
> wouldn't make much sense, but thank you for confirming that setting
> memtune->hard_limit adjusts both the limits to the requested value.

Right, whenever libvirt needs to set a memory locking limit for zero
copy migration it uses memtune->hard_limit if it is set (RDMA migration
does the same), otherwise the configured memory amount for the domain is
used.

Anyway, unless you have a specific reason for setting the hard_limit,
it's usually better to leave it unset as the overall memory consumption
of a domain including the memory consumed by QEMU is hard to predict and
you would risk the domain to be killed unexpectedly by the kernel when
the limit is reached.

Jirka