[libvirt] [V4] RFC for support cache tune(CAT) in libvirt

[乔立勇(Eli Qiao)]

hi all


Sorry for resend this again, I forget to add subject :( ...


Thanks all for the comment for previous RFC version, I summary all input
here also with Opens which I am not sure, can you please review again?



#Propose Changes



## Expose cache information into capabilities





Extend capabilities to expose all level cache resource information and
should tell the topology of caches also tell which kinds of resources can
be tuned/allocation



There information comes from
/sys/devices/system/cpu/cpu0/cache/index${level}/size
also /sys/fs/resctrl/ (this is CAT sys interface in linux kernel)



virsh capabilities



    <cache>

      <bank id='0, 'type="l3" size="56320" units="KiB" cpus="0,1,2,6,7,8"/>
 <--------------------- level 3 cache is per socket, so group them by
socket id

           <control unit="KiB" min="2816"/>

      <bank id='1', type="l3" size="56320" units="KiB"
cpus="3,4,5,9,10,11"/>

      <bank id='2' type="l2" size="256" units="KiB" cpus="0"/>

      <bank id='3' type="l2" size="256" units="KiB" cpus="1"/>

      <bank id='4' type="l2" size="256" units="KiB" cpus="2"/>

      <bank id='5' type="l2" size="256" units="KiB" cpus="3"/>

      <bank id='6' type="l2" size="256" units="KiB" cpus="4"/>

...

     <cache>



Opens

1.      how about add socket id to bank for bank type = l3 ?

2.      do we really want to expose l2/l3 cache for now , they are per core
resource and linux kernel don't support l2 yet (depend no hardware)?

3.      if enable CDP in resctrl, for bank type=l3 , it will be split to
l3data l3code, should expose this ability.

      <bank type="l3" size="56320" units="KiB" cpus="0,1,2,6,7,8"/>
 <--------------------- level 3 cache is per socket, so group them by
socket id

           <control unit="KiB" min="2816" cdp="enabled"/>



## Provide a new API to get the avail cache on each bank, such as the
output are:



   id=0

   type=l3

   avail=56320

   total = ?? <--------- do we need this?



   id=1

   type=l3

   avail=56320



   id=3

   type=l2

   avail=256



Opens:

·         Don't expose the avail cache information if the host can not do
the allocation of that type cache(eg, for l2 currently) ?

·         We can not make all of the cache , the reservation amount is the
min_cbm_len (=1) * min_unit .

·         do we need to expose total?







##  enable CAT for a domain



1 Domain XML changes



   <cputune>

       <cache id="1" host_id="0" type="l3" size="5632" unit="KiB"/>

       <cache id="2" host_id="1" type="l3" size="5632" unit="KiB"/>



       <cpu_cache vcpus="0-3" id="1"/>

       <cpu_cache vcpus="4-7" id="2"/>

       <iothread_cache iothreads="0-1" id="1"/>

       <emulator_cache id="2"/>

   </cputune>



2. Extend cputune command ?





Opens:



1. Do we accept to extend existed API ? or using new API/virsh?

2. How to calculate cache size -> CBM bit?



eg:

5632/ 2816 = 2 bits

5733/ 2816 = 2 bits or 3 bits?







## Restriction for using cache tune on multiple sockets' host.



The l3 cache is per socket resource, kernel need to know about what's
affinity looks like, so for a VM which running on a multiple socket's host,
it should have NUMA setting or vcpuset pin setting. Or cache tune will fail.



[1] kernel support https://git.kernel.org/cgit/linux/kernel/git/tip/tip
.git/tree/arch/x86/kernel/cpu/intel_rdt.c?h=x86/cache



[2] libvirt PoC(not finished yet) https://github.com/taget/
libvirt/commits/cat_new

-- 
Best regards
- Eli

天涯无处不重逢
a leaf duckweed belongs to the sea , where not to meet in life
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/libvir-list/attachments/20170113/5c403684/attachment-0001.htm>