[libvirt] [PATCH 00/10] Introduce x86 Cache Monitoring Technology (CMT)
Huaqiang,Wang
huaqiang.wang at intel.com
Wed Sep 5 01:39:25 UTC 2018
hi reviewer,
I understand libvirt community is quite active and you are quite busy. I
am written here to know if you ever noticed this patch series, and
welcome your comment.
BR
On 2018年08月27日 19:23, Wang Huaqiang wrote:
> This series of patches introduced the x86 Cache Monitoring Technology
> (CMT) to libvirt by interacting with kernel resource control (resctrl)
> interface. CMT is one of the Intel(R) x86 CPU feature which belongs to
> the Resource Director Technology (RDT). CMT reports the occupancy of the
> last level cache, which is shared by all CPU cores.
>
> We have serval discussion about the enabling of CMT, please refer to
> following links for the RFCs.
> RFCv3
> https://www.redhat.com/archives/libvir-list/2018-August/msg01213.html
> RFCv2
> https://www.redhat.com/archives/libvir-list/2018-July/msg00409.html
> https://www.redhat.com/archives/libvir-list/2018-July/msg01241.html
> RFCv1
> https://www.redhat.com/archives/libvir-list/2018-June/msg00674.html
>
> 1. About reason why CMT is necessary in libvirt?
> The perf events of 'CMT, MBML, MBMT' have been phased out since Linux
> kernel commit c39a0e2c8850f08249383f2425dbd8dbe4baad69, in libvirt
> the perf based cmt,mbm will not work with the latest linux kernel. These
> patches add CMT feature to libvirt through kernel resctrlfs interface.
>
> 2. Interfaces for CMT from the high level.
>
> 2.1 Query the host capability of CMT.
>
> The element 'monitor' represents the host capabilities of CMT.
> The explanations of involved CMT attributes:
> - 'maxAllocs' denotes the maximum monitoring groups could be created,
> which is limited by the number of hardware 'RMID'.
> - 'threshold' denotes the upper bound of cache occupancy for current
> group, in bytes, to determine if an RMID can be reused.
> - element 'feature' denotes the monitoring feature supported.
> - 'llc_occupancy' is the feature for reporting the last level cache
> occupancy information.
>
> # virsh capabilities
> ...
> <cache>
> <bank id='0' level='3' type='both' size='15' unit='MiB' cpus='0-5'>
> <control granularity='768' unit='KiB' type='code' maxAllocs='8'/>
> <control granularity='768' unit='KiB' type='data' maxAllocs='8'/>
> + <monitor threshold='540672' unit='B' maxAllocs='176'/>
> + <feature name=llc_occupancy/>
> + </monitor>
> </bank>
> <bank id='1' level='3' type='both' size='15' unit='MiB' cpus='6-11'>
> <control granularity='768' unit='KiB' type='code' maxAllocs='8'/>
> <control granularity='768' unit='KiB' type='data' maxAllocs='8'/>
> + <monitor threshold='540672' unit='B' maxAllocs='176'/>
> + <feature name=llc_occupancy/>
> + </monitor>
> </bank>
> </cache>
> ...
>
> 2.2 Create cache monitoring group (cache monitor).
>
> The main interface for creating monitoring group is through XML file. The
> proposed configuration is like:
>
> <cputune>
> <cachetune vcpus='1'>
> <cache id='0' level='3' type='code' size='7680' unit='KiB'/>
> <cache id='1' level='3' type='data' size='3840' unit='KiB'/>
> + <monitor vcpus='1'/>
> </cachetune>
> <cachetune vcpus='4-7'>
> + <monitor vcpus='4-6'/>
> </cachetune>
> </cputune>
>
> In above XML, created 2 cache resctrl allocation groups and 2 resctrl
> monitoring groups.
> The changes of cache monitor will be effective in next booting of VM.
>
> 2.3 Show CMT result through command 'domstats'
>
> Adding the interface in qemu to report this information for resource
> monitor group through command 'virsh domstats --cpu-total'.
> Below is a typical output:
>
> # virsh domstats 1 --cpu-total
> Domain: 'ubuntu16.04-base'
> ...
> cpu.cache.monitor.count=2
> cpu.cache.0.name=vcpus_1
> cpu.cache.0.vcpus=1
> cpu.cache.0.bank.count=2
> cpu.cache.0.bank.0.id=0
> cpu.cache.0.bank.0.bytes=4505600
> cpu.cache.0.bank.1.id=1
> cpu.cache.0.bank.1.bytes=5586944
> cpu.cache.1.name=vcpus_4-6
> cpu.cache.1.vcpus=4,5,6
> cpu.cache.1.bank.count=2
> cpu.cache.1.bank.0.id=0
> cpu.cache.1.bank.0.bytes=17571840
> cpu.cache.1.bank.1.id=1
> cpu.cache.1.bank.1.bytes=29106176
>
>
>
>
> **Changes Since RFCv3**
> In the output of 'domstats', added
> 'cpu.cache.<cmt_group_index>.bank.<bank_index>.id'
> to tell the OS assigned cache bank id of current cache.
> Changes is prefixed with a '+':
>
> # virsh domstats 1 --cpu-total
> Domain: 'ubuntu16.04-base'
> ...
> cpu.cache.monitor.count=2
> cpu.cache.0.name=vcpus_1
> cpu.cache.0.vcpus=1
> cpu.cache.0.bank.count=2
> + cpu.cache.0.bank.0.id=0
> cpu.cache.0.bank.0.bytes=4505600
> + cpu.cache.0.bank.1.id=1
> cpu.cache.0.bank.1.bytes=5586944
> cpu.cache.1.name=vcpus_4-6
> cpu.cache.1.vcpus=4,5,6
> cpu.cache.1.bank.count=2
> + cpu.cache.1.bank.0.id=0
> cpu.cache.1.bank.0.bytes=17571840
> + cpu.cache.1.bank.1.id=1
> cpu.cache.1.bank.1.bytes=29106176
>
> Wang Huaqiang (10):
> conf: Renamed 'controlBuf' to 'childrenBuf'
> util: add interface retrieving CMT capability
> conf: Add CMT capability to host
> test: add test case for resctrl monitor
> util: resctrl: refactoring some functions
> util: Introduce resctrl monitor for CMT
> conf: refactor virDomainResctrlAppend
> conf: introduce resctrl monitor group in domain
> qemu: Introduce resctrl monitoring group
> qemu: Report cache occupancy (CMT) with domstats
>
> .gnulib | 1 -
> docs/formatdomain.html.in | 14 +-
> docs/schemas/capability.rng | 28 +
> docs/schemas/domaincommon.rng | 11 +-
> src/conf/capabilities.c | 51 +-
> src/conf/capabilities.h | 1 +
> src/conf/domain_conf.c | 159 +++++-
> src/conf/domain_conf.h | 20 +
> src/libvirt-domain.c | 9 +
> src/libvirt_private.syms | 6 +
> src/qemu/qemu_driver.c | 265 ++++++++-
> src/qemu/qemu_process.c | 40 +-
> src/util/virresctrl.c | 597 +++++++++++++++++++--
> src/util/virresctrl.h | 48 +-
> tests/genericxml2xmlindata/cachetune-cdp.xml | 2 +
> .../cachetune-colliding-monitors.xml | 36 ++
> tests/genericxml2xmlindata/cachetune-small.xml | 1 +
> tests/genericxml2xmlindata/cachetune.xml | 3 +
> tests/genericxml2xmltest.c | 4 +
> .../resctrl/info/L3_MON/max_threshold_occupancy | 1 +
> .../linux-resctrl/resctrl/info/L3_MON/mon_features | 3 +
> .../linux-resctrl/resctrl/info/L3_MON/num_rmids | 1 +
> tests/vircaps2xmldata/vircaps-x86_64-resctrl.xml | 6 +
> 23 files changed, 1208 insertions(+), 99 deletions(-)
> delete mode 160000 .gnulib
> create mode 100644 tests/genericxml2xmlindata/cachetune-colliding-monitors.xml
> create mode 100644 tests/vircaps2xmldata/linux-resctrl/resctrl/info/L3_MON/max_threshold_occupancy
> create mode 100644 tests/vircaps2xmldata/linux-resctrl/resctrl/info/L3_MON/mon_features
> create mode 100644 tests/vircaps2xmldata/linux-resctrl/resctrl/info/L3_MON/num_rmids
>
More information about the libvir-list
mailing list