<div dir="ltr">hi all Sorry for resend this again, I forget to add subject :( ... Thanks all for the comment for previous RFC version, I summary all input here also with Opens which I am not sure, can you please review again? #Propose Changes ## Expose cache information into capabilities Extend capabilities to expose all level cache resource information and should tell the topology of caches also tell which kinds of resources can be tuned/allocation There information comes from /sys/devices/system/cpu/cpu0/cache/index${level}/size also /sys/fs/resctrl/ (this is CAT sys interface in linux kernel) virsh capabilities <cache> <bank id='0, 'type="l3" size="56320" units="KiB" cpus="0,1,2,6,7,8"/> <--------------------- level 3 cache is per socket, so group them by socket id <control unit="KiB" min="2816"/> <bank id='1', type="l3" size="56320" units="KiB" cpus="3,4,5,9,10,11"/> <bank id='2' type="l2" size="256" units="KiB" cpus="0"/> <bank id='3' type="l2" size="256" units="KiB" cpus="1"/> <bank id='4' type="l2" size="256" units="KiB" cpus="2"/> <bank id='5' type="l2" size="256" units="KiB" cpus="3"/> <bank id='6' type="l2" size="256" units="KiB" cpus="4"/> ... <cache> <div style="font-size:14px"> Opens1. how about add socket id to bank for bank type = l3 ? 2. do we really want to expose l2/l3 cache for now , they are per core resource and linux kernel don't support l2 yet (depend no hardware)?3. if enable CDP in resctrl, for bank type=l3 , it will be split to l3data l3code, should expose this ability.<div class="gmail-m_-2643208640856821907gmail-yj6qo gmail-m_-2643208640856821907gmail-ajU" style="margin:2px 0px 0px"><div id="gmail-m_-2643208640856821907gmail-:1y9" class="gmail-m_-2643208640856821907gmail-ajR"><img class="gmail-m_-2643208640856821907gmail-ajT gmail-CToWUd" src="https://ssl.gstatic.com/ui/v1/icons/mail/images/cleardot.gif"></div></div><div class="gmail-m_-2643208640856821907gmail-adL"><div class="gmail-m_-2643208640856821907gmail-adm"></div><div class="gmail-m_-2643208640856821907gmail-im"> <bank type="l3" size="56320" units="KiB" cpus="0,1,2,6,7,8"/> <--------------------- level 3 cache is per socket, so group them by socket id <control unit="KiB" min="2816" cdp="enabled"/> ## Provide a new API to get the avail cache on each bank, such as the output are: id=0 type=l3 avail=56320 total = ?? <--------- do we need this? id=1 type=l3 avail=56320 id=3 type=l2 avail=256 Opens:· Don't expose the avail cache information if the host can not do the allocation of that type cache(eg, for l2 currently) ?· We can not make all of the cache , the reservation amount is the min_cbm_len (=1) * min_unit . · do we need to expose total? ## enable CAT for a domain 1 Domain XML changes <cputune> <cache id="1" host_id="0" type="l3" size="5632" unit="KiB"/> <cache id="2" host_id="1" type="l3" size="5632" unit="KiB"/> <cpu_cache vcpus="0-3" id="1"/> <cpu_cache vcpus="4-7" id="2"/> <iothread_cache iothreads="0-1" id="1"/> <emulator_cache id="2"/> </cputune> 2. Extend cputune command ? Opens: 1. Do we accept to extend existed API ? or using new API/virsh?2. How to calculate cache size -> CBM bit? eg: 5632/ 2816 = 2 bits5733/ 2816 = 2 bits or 3 bits? ## Restriction for using cache tune on multiple sockets' host. The l3 cache is per socket resource, kernel need to know about what's affinity looks like, so for a VM which running on a multiple socket's host, it should have NUMA setting or vcpuset pin setting. Or cache tune will fail. [1] kernel support <a href="https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git/tree/arch/x86/kernel/cpu/intel_rdt.c?h=x86/cache" target="_blank">https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git/tree/arch/x86/kernel/cpu/intel_rdt.c?h=x86/cache</a> [2] libvirt PoC(not finished yet) <a href="https://github.com/taget/libvirt/commits/cat_new" target="_blank">https://github.com/taget/libvirt/commits/cat_new</a></div></div></div><div> </div>-- <div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>Best regards </div><div>- Eli</div><div> </div><div>天涯无处不重逢</div><div>a leaf duckweed belongs to the sea , where not to meet in life </div></div></div></div></div> </div>