[Libvir] The problem of the definition of tuning informations

[Ryan Harper]

* Daniel Veillard <veillard at redhat.com> [2007-11-08 15:27]:
> On Thu, Nov 08, 2007 at 02:00:10PM -0600, Ryan Harper wrote:
> > * Daniel Veillard <veillard at redhat.com> [2007-11-08 10:08]:
> > >   I promised that mail for the beginning of the week but I still have
> > > I think tuning informations are that set of parameters associated
> > > to a domain or a host, which are not stricly needed to get the 
> > > domain(s) working but improve their runtime behaviour.
> > > To me this includes:
> > >    - scheduling parameters the scope may be host/hypervisor/domain
> > >    - vcpu affinity i.e. to which set of physical CPU each of the
> > >      vcpu may be bound
> > >    - and possibly others ...
> > > 
> > > The problem:
> > > ------------
> > > People would like to associate those to the XML domain informations,
> > > the goal being to be able to restore those informations when a domain
> > > (re-)starts. 
> > > I have been objecting it so far because, I think those informations
> > > don't have the same lifetime and scope as the other domain informations
> > > saved in the XML. Since they are not needed to start the domain, and
> > > that once the domain is started the existing domain API can be used
> > > to change those informations, it is better to keep them separate.
> > 
> > For at least (maybe only) Xen NUMA systems, the application of "tuning"
> > information after a domain is started does not achieve the same affect
> > as including the information during the initial construction of the
> > domain.  In particular, Xen needs to know which physical cpus are being
> > used to determine which cpus it from which numanode it will allocate
> > memory.  Adjusting affinity after the domain has allocated memory
> > doesn't allow libvirt or any management app to control from which node
> > domains pull memory.
> 
>   yes, I understand and that's why I agreed to add the cpuset information
> at that point it's more than tunning because it may be irreversible for the
> lifetime of the domain, so this really should be in the XML. I'm not
> suggesting to go back about 'cpu affinity' i.e. to which physical CPUs
> a domain should be bound, but 'vcpu affinity' i.e. then how the virtual
> CPUs of the domain are mapped onto that cpu set, that can change

OK, I see your distinction here.

> dynamically without (serious) performance penalty. 

At least for Xen, the 'cpu' affinity specified with a domain is
only accessible via the xen config file and is not enforced in any way
such that it prevents from someone "tuning" a domain to use physical
cpus outside of the specified cpumap.  Users can can certainly
specify a cpu outside of the original cpuset from the config file which
in a NUMA scenario has the potential for serious performance penalties.

> 
> > I don't have any objection to separating "tuning" information as long as
> > we have the ability to merge permanent domain parameters with its
> > "tuning" information prior to domain construction.
> 
>   My point is that you don't need the tuning informations to create the
> domain, if you need them it's not tuning. When you say you want to
> merge them, do you want this to create the domain ? It should not
> be necessary (or I take a counter example that would help me), right ?

I agree here.  I was lumping cpuset info into your tunable category but
you clarified the distinction above.  I just want to ensure that initial
cpuset mapping is present prior to constructing a domain as that is
integral for proper Xen NUMA memory allocation.


-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh at us.ibm.com