[libvirt] [RFC] Memory controller exploitation in libvirt

[Balbir Singh]

On Mon, Aug 30, 2010 at 11:56 AM, Nikunj A. Dadhania
<nikunj at linux.vnet.ibm.com> wrote:
> On Tue, 24 Aug 2010 11:07:29 +0100, "Daniel P. Berrange" <berrange at redhat.com> wrote:
>> On Tue, Aug 24, 2010 at 03:17:44PM +0530, Nikunj A. Dadhania wrote:
>> >
>> > On Tue, 24 Aug 2010 11:02:49 +0200, Matthias Bolte <matthias.bolte at googlemail.com> wrote:
>> >
>> > <snip>
>> >
>> > > Yes the ESX driver allows to control ballooning through
>> > > virDomainSetMemory and virDomainSetMaxMemory.
>> > >
>> > > ESX itself also allows to set what's called memoryMinGaurantee in the
>> > > thread, but this is not exposed in libvirt.
>> > LXC driver is using virDomainSetMemory to set the memory hard limit while
>> > QEmu/ESX use them to change the ballooning. And as you said, ESX does support
>> > memoryMinGaurantee, we can get this exported in libvirt using this new API.
>> >
>> > Here I am trying to group all the memory related parameters into one single
>> > public API as we have in virDomainSetSchedulerParameters. Currently, the names
>> > are not conveying what they modify in the below layer and are confusing.
>>
>> For historical design record, I think it would be good to write a short
>> description of what memory tunables are available for each hypervisor,
>> covering VMWare, OpenVZ, Xen, KVM and LXC (the latter both cgroups based).
>> I do recall that OpenVZ  in particular had a huge number of memory
>> tunables.
>>
> This is an attempt at covering the memory tunables supported by various
> hypervisors in libvirt. Let me know if I have missed any memory
> tunable. Moreover, inputs from the maintainers/key contributes of each HVs on
> these parameters is appreciable. This would help in getting a complete
> coverage of the memory tunables that libvirt can support.
>
> 1) OpenVZ
> =========
> vmguarpages: Memory allocation guarantee, in pages.
> kmemsize: Size of unswappable kernel memory(in bytes), allocated for processes in this
>          container.
> oomguarpages: The guaranteed amount of memory for the case the memory is
>              “over-booked” (out-of-memory kill guarantee), in pages.
> privvmpages: Memory allocation limit, in pages.
>
>
> OpenVZ driver does not implement any of these functions:
> domainSetMemory
> domainSetMaxMemory
> domainGetMaxMemory
>
> Although, the driver has an internal implementation for setting memory:
> openvzDomainSetMemoryInternal that is read from the domain xml file.
>
> 2) VMWare
> =========
> ConfiuredSize:  Virtual memory the guest can have.
>
> Shares: Priority of the VM, in case there is not enough memory or in case when
>        there is more memory. It has symbolic values like Low, Normal, High
>        and Custom
>
> Reservation: Gauranteed lower bound on the amount of the physical memory that
>             the host reserves for the VM even in case of the overcommit. The
>             VM is allowed to allocate till this level and after it has hit
>             the reservation, those pages are not reclaimed. In case, if guest
>             is not using till the reservation, the host can use that portion
>             of memory.
>
> Limit: This is the upper bound for the amount of physical memory that the host
>       can allocate for the VM.
>
> Memory Balloon
>
> ESX driver uses following:
> * domainSetMaxMemory to set the max virtual memory for the VM.
> * domainSetMemory to inflate/deflate the balloon.
> * ESX provides lower bound(Reservation), but is not being exploited currently.
>
> 3) Xen
> ======
> maxmem_set: Maximum amount of memory reservation of the domain
> mem_target_set: Set current memory usage of the domain
>
>
> 4) KVM & LXC
> ============
> memory.limit_in_bytes: Memory hard limit
> memory.soft_limit_in_bytes: Memory limit held during contention

"held" might not be the right word for soft limit.

> memory.memsw_limit_in_bytes: Memory+Swap hard limit
> memory.swapiness: Controls the tendency of moving the VM processes to the
>                  swap. Value range is 0-100, where 0 means, avoid swapping as
>                  long as possible and 100 means aggressively swap processes.
>
> Statistics:
> memory.usage_in_bytes: Current memory usage
> memory.memsw_usage_in_bytes: Current memory+swap usage
> memory.max_usage_in_bytes: Maximum memory usage recorded
> memory.memsw_max_usage_in_bytes: Maximum memory+swap usage

We also have memory.stat, memory.hierarchy - The question is do we
care about hierarchical control? We also have controls to decide
whether to move memory on moving from one cgroup to another, that
might not apply to the LXC/QEMU case. There is also memory.failcnt
which I am not sure makes sense to export.

Balbir