[libvirt] [RFC] Memory controller exploitation in libvirt

[Nikunj A. Dadhania]

 Subject: [RFC] Memory controller exploitation in libvirt

 Memory CGroup is a kernel feature that can be exploited effectively in the
 current libvirt/qemu driver. Here is a shot at that.

 At present, QEmu uses memory ballooning feature, where the memory can be
 inflated/deflated as and when needed, co-operatively between the host and
 the guest. There should be some mechanism where the host can have more
 control over the guests memory usage. Memory CGroup provides features such
 as hard-limit and soft-limit for memory, and hard-limit for swap area.

 Design 1: Provide new API and XML changes for resource management
 =================================================================

 All the memory controller tunables are not supported with the current
 abstractions provided by the libvirt API. libvirt works on various OS. This
 new API will support GNU/Linux initially and as and when other platforms
 starts supporting memory tunables, the interface could be enabled for
 them. Adding following two function pointer to the virDriver interface.

 1) domainSetMemoryParameters: which would take one or more name-value
    pairs. This makes the API extensible, and agnostic to the kind of
    parameters supported by various Hypervisors.
 2) domainGetMemoryParameters: For getting current memory parameters

 Corresponding libvirt public API:
 int virDomainSetMemoryParamters (virDomainPtr domain, 
     				  virMemoryParamterPtr params, 
      		                  unsigned int nparams);
 int virDomainGetMemoryParamters (virDomainPtr domain, 
     				  virMemoryParamterPtr params, 
      		                  unsigned int nparams);



 Parameter list supported:

    MemoryHardLimits (memory.limits_in_bytes) - Maximum memory 
    MemorySoftLimits (memory.softlimit_in_bytes) - Desired memory 
    MemoryMinimumGaurantee - Minimum memory required (without this amount of
    memory, VM should not be started) 

    SwapHardLimits (memory.memsw_limit_in_bytes) - Maximum swap 
    SwapSoftLimits (Currently not supported by kernel) - Desired swap space 

    Tunables memory.limit_in_bytes, memory.softlimit_in_bytes and
    memory.memsw_limit_in_bytes are provided by the memory controller in the
    Linux kernel.
    
 I am not an expert here, so just listing what new elements need to be added
 to the XML schema:

 <define name="resource">
    <element memory>
    	<element memoryHardLimit/>
    	<element memorySoftLimit/>
        <element memoryMinGaurantee/>
        <element swapHardLimit/>
        <element swapSoftLimit/>
    </element>
 </define>

 Pros:
 * Support all the tunables exported by the kernel
 * More tunables can be added as and when required
 
 Cons:
 * Code changes would touch various levels
 * Might need to redefine(changing the scope) of existing memory
   API. Currently, domainSetMemory is used to set limit_in_bytes in LXC and
   memory ballooning in QEmu. While the domainSetMaxMemory is not defined in
   QEmu and in case of LXC it is setting the internal object's maxmem
   variable.

 Future: 

 * Later on, CPU/IO/Network controllers related tunables can be
   added/enhanced along with the APIs/XML elements:

         CPUHardLimit
    	 CPUSoftLimit
    	 CPUShare
    	 CPUPercentage
    	 IO_BW_Softlimit
    	 IO_BW_Hardlimit
    	 IO_BW_percentage

 * libvirt-cim support for resource management

 Design 2: Reuse the current memory APIs in libvirt
 ==================================================

 Use memory.limit_in_bytes to tweak memory hard limits 
 Init - Set the memory.limit_in_bytes to maximum mem.

 Claiming memory from guest:
 a) Reduce balloon size
 b) If the guest does not co-operate(How do we know?), reduce
 memory.limit_in_bytes. 

 Allocating memory more than max memory: How to solve this? As we have
 already set the max balloon size. We can only play within this!

 Pros:
 * Few changes
 * Is not intrusive

 Cons:
 * SetMemory and SetMaxMemory usage is confusing.
 * SetMemory is too generic a name, it does not cover all the tunables.
 * Does not support memory softlimit
 * Does not have support to reserve the memory swap region
 * This solution is not extensible

IMO, "Design 1" is more generic and extensible for various memory
tuneables. 

Nikunj