[libvirt] [RFC] Memory controller exploitation in libvirt

[Balbir Singh]

* Nikunj A. Dadhania <nikunj at linux.vnet.ibm.com> [2010-08-24 11:53:27]:

> 
>  Subject: [RFC] Memory controller exploitation in libvirt
> 
>  Memory CGroup is a kernel feature that can be exploited effectively in the
>  current libvirt/qemu driver. Here is a shot at that.
> 
>  At present, QEmu uses memory ballooning feature, where the memory can be
>  inflated/deflated as and when needed, co-operatively between the host and
>  the guest. There should be some mechanism where the host can have more
>  control over the guests memory usage. Memory CGroup provides features such
>  as hard-limit and soft-limit for memory, and hard-limit for swap area.
> 
>  Design 1: Provide new API and XML changes for resource management
>  =================================================================
> 
>  All the memory controller tunables are not supported with the current
>  abstractions provided by the libvirt API. libvirt works on various OS. This
>  new API will support GNU/Linux initially and as and when other platforms
>  starts supporting memory tunables, the interface could be enabled for
>  them. Adding following two function pointer to the virDriver interface.
> 
>  1) domainSetMemoryParameters: which would take one or more name-value
>     pairs. This makes the API extensible, and agnostic to the kind of
>     parameters supported by various Hypervisors.
>  2) domainGetMemoryParameters: For getting current memory parameters
> 
>  Corresponding libvirt public API:
>  int virDomainSetMemoryParamters (virDomainPtr domain, 
>      				  virMemoryParamterPtr params, 
>       		                  unsigned int nparams);
>  int virDomainGetMemoryParamters (virDomainPtr domain, 
>      				  virMemoryParamterPtr params, 
>       		                  unsigned int nparams);
> 
> 

Does nparams imply setting several parameters together? Does bulk
loading help? I would prefer splitting out the API if possible
into

        virCgroupSetMemory() - already present in src/util/cgroup.c
        virCgroupGetMemory() - already present in src/util/cgroup.c
        virCgroupSetMemorySoftLimit()
        virCgroupSetMemoryHardLimit()
        virCgroupSetMemorySwapHardLimit()
        virCgroupGetStats()

> 
>  Parameter list supported:
> 
>     MemoryHardLimits (memory.limits_in_bytes) - Maximum memory 
>     MemorySoftLimits (memory.softlimit_in_bytes) - Desired memory 

Soft limits allows you to set memory limit on contention.

>     MemoryMinimumGaurantee - Minimum memory required (without this amount of
>     memory, VM should not be started) 
> 
>     SwapHardLimits (memory.memsw_limit_in_bytes) - Maximum swap 
>     SwapSoftLimits (Currently not supported by kernel) - Desired swap space 
> 

We *dont* support SwapSoftLimits in the memory cgroup controller with
no plans to support it in the future either at this point. The
semantics are just too hard to get right at the moment.

>     Tunables memory.limit_in_bytes, memory.softlimit_in_bytes and
>     memory.memsw_limit_in_bytes are provided by the memory controller in the
>     Linux kernel.
>     
>  I am not an expert here, so just listing what new elements need to be added
>  to the XML schema:
> 
>  <define name="resource">
>     <element memory>
>     	<element memoryHardLimit/>
>     	<element memorySoftLimit/>
>         <element memoryMinGaurantee/>
>         <element swapHardLimit/>
>         <element swapSoftLimit/>
>     </element>
>  </define>
> 

I'd prefer a syntax that integrates well with what we currently have

        <cgroup>
                <path>...</path>
                <controller>
                <name>..</name>
                <soft limit>...</>
                <hard limit>...</>
                </controller>
        ...
        </cgroup>

But I am not an XML expert or an export in designing XML
configurations.

>  Pros:
>  * Support all the tunables exported by the kernel
>  * More tunables can be added as and when required
> 
>  Cons:
>  * Code changes would touch various levels
>  * Might need to redefine(changing the scope) of existing memory
>    API. Currently, domainSetMemory is used to set limit_in_bytes in LXC and
>    memory ballooning in QEmu. While the domainSetMaxMemory is not defined in
>    QEmu and in case of LXC it is setting the internal object's maxmem
>    variable.
> 
>  Future: 
> 
>  * Later on, CPU/IO/Network controllers related tunables can be
>    added/enhanced along with the APIs/XML elements:
> 
>          CPUHardLimit
>     	 CPUSoftLimit
>     	 CPUShare
>     	 CPUPercentage
>     	 IO_BW_Softlimit
>     	 IO_BW_Hardlimit
>     	 IO_BW_percentage
> 
>  * libvirt-cim support for resource management
> 
>  Design 2: Reuse the current memory APIs in libvirt
>  ==================================================
> 
>  Use memory.limit_in_bytes to tweak memory hard limits 
>  Init - Set the memory.limit_in_bytes to maximum mem.
> 
>  Claiming memory from guest:
>  a) Reduce balloon size
>  b) If the guest does not co-operate(How do we know?), reduce
>  memory.limit_in_bytes. 


This is a policy and I am not sure an API should be hiding policy
succintly

> 
>  Allocating memory more than max memory: How to solve this? As we have
>  already set the max balloon size. We can only play within this!
> 
>  Pros:
>  * Few changes
>  * Is not intrusive
> 
>  Cons:
>  * SetMemory and SetMaxMemory usage is confusing.
>  * SetMemory is too generic a name, it does not cover all the tunables.
>  * Does not support memory softlimit
>  * Does not have support to reserve the memory swap region
>  * This solution is not extensible
> 
> IMO, "Design 1" is more generic and extensible for various memory
> tuneables. 
> 

Agreed, 1 is better.

> Nikunj

-- 
	Three Cheers,
	Balbir