[libvirt] [RFC PATCH 0/3] RFC for X86 RDT Cache Monitoring Technology (CMT) support
Martin Kletzander
mkletzan at redhat.com
Mon Jun 11 08:29:49 UTC 2018
[It would be nice if you wrapped the long lines]
On Fri, Jun 08, 2018 at 05:02:16PM +0800, Wang Huaqiang wrote:
>This is an RFC request for supporting CPU Cache Monitoring Technology (CMT) feature in libvirt. Since MBM is also another feature which is very close to CMT, for simplicity we only discuss CMT here. MBM is the followup that will be implemented after CMT.
>About CMT please refer to Intel x86 SDM section 17.18 of volume 3 (link:https://software.intel.com/en-us/articles/intel-sdm).
>
Can you elaborate on how is this different to the CMT perf event that is already
in libvirt and can be monitored through domstats API?
https://libvirt.org/formatdomain.html#elementsPerf
>## About '_virResctrlMon' interface
>
>The cache allocation technology (CAT) has already been implemented in util/virresctrl.* which interacts with Linux kernel resctrl file system. Very simlimar to CAT, the CMT object is represented by 'struct _virResctrlMon', which is
>
>```
>struct _virResctrlMon {
> virObject parent;
>
> /* pairedalloc: pointer to a resctrl allocaion it paried with.
> * NULL for a resctrl monitoring group not associated with
> * any allocation. */
> virResctrlAllocPtr pairedalloc;
> /* The identifier (any unique string for now) */
> char *id;
> /* libvirt-generated path, may be identical to alloction path
> * may not if allocation is ready */
> char *path;
>};
>```
>
>Almost following the same logic behind '_virResctrlAlloc' which is mainly presented in file 'virresctrl.c', a group of APIs has been designed to manipulate '_virResctrlMon'. The '_virResctrlMon' shares a lot in common with '_virResctrlAlloc' except field 'pairedalloc'.
>'pairedalloc' stores the pointer of paired resctrl allocation object. With current libvirt resctrl implementation, if a resctrl '_virResctrlAlloc' object is created, the CMT hardware is enabled automatically and shares the same folder under same resctrlfs, I call the CMT '_virResctrlMon' object that shares the same folder under resctrlfs as 'paired' _virResctrlMon, further, one '_virResctrlMon' and one '_virResctrlAlloc' are a pare. In '_virResctrlMon' the paired '_virResctrlAlloc' is tracked through pairedalloc. paired mon group could not be dynamically enabled or disabled during runtime.
>'pairedalloc' could be set to NULL, which creates a non-paired mon group object. Which is necessory because CMT could work independently to monitor the utilization of critical CPU resouces (cache or memory bandwidth) without allocating any dedicated cache or memory bandwidth. A non-paired mon group object represents an independent working CMT. Non-paired mon group could be enabled or disabled during runtime.
>
>## About virsh command 'resctrl'
>
>To set or get the resctrl mon group (hardware CMT), a virsh command 'resctrl' is created. here are the common usages:
The command does make sense for people who know how the stuff works on the
inside or have seen the code in libvirt. For other users the name 'resctrl' is
going to feel very much arbitrary. We re trying to abstract the details for
users, so I don't see why it should be named 'resctrl' when it handles "RDT
Monitoring Status".
>```
>[root at dl-c200 david]# virsh list --all
> Id Name State
>----------------------------------------------------
> 1 vm3 running
> 3 vm2 running
> - vm1 shut off
>```
>
>### Test on a running domain vm3
>To get RDT monitoring status, type 'virsh resctrl <domain>'
>```
> [root at dl-c200 david]# virsh resctrl vm3
> RDT Monitoring Status: Enabled
>```
>
>To enable RDT monitoring, type 'virsh resctrl <domain> --enable'
>```
> [root at dl-c200 david]# virsh resctrl vm3 --enable
> RDT Monitoring Status: Enabled
>```
>
>To diable RDT monitoring, type 'virsh resctrl <domain> --disable'
>```
> [root at dl-c200 david]# virsh resctrl vm3 --disable
> RDT Monitoring Status: Disabled
>
> [root at dl-c200 david]# virsh resctrl vm3
> RDT Monitoring Status: Disabled
>```
>
>### test on domain not running vm1
>if domain is not active, it will fail to set RDT monitoring status, and also get the state of 'disabled'
>```
> [root at dl-c200 david]# virsh resctrl vm1
> RDT Monitoring Status: Disabled
>
> [root at dl-c200 david]# virsh resctrl vm1 --enable
> error: Requested operation is not valid: domain is not running
>
> [root at dl-c200 david]# virsh resctrl vm1 --disable
> error: Requested operation is not valid: domain is not running
>```
>
Can't these commands enable it in the XML? It would be nice if the XML part was
shown here in the explanation.
>### test on domain vm2
>domain vm2 is active and the CAT functionality is enabled through 'cachetune' (configured in 'cputune/cachetune' section). So the resctrl mon group is a 'paried' one, for 'pared' mon group, the RDT monitoring could not be disabled. If it is allowed to disable 'paire' mon group, we have to destroy resctrl allocation folders which is not supported by current cache allocation design.
What if you have multiple cachetunes? What if the cachetune is only set for one
vcpu and you want to monitor the others as well? I guess I have to see the
patches to understand why you have so much information stored for something that
looks like a boolean (enable/disable).
>```
> [root at dl-c200 libvirt]# virsh resctrl vm2 --enable
> RDT Monitoring Status: Enabled (forced by cachetune)
>
> [root at dl-c200 libvirt]# virsh resctrl vm2 --disable
> RDT Monitoring Status: Enabled (forced by cachetune)
>
> [root at dl-c200 libvirt]# virsh resctrl vm2
> RDT Monitoring Status: Enabled (forced by cachetune)
>```
>
>## About showing the utilization information of RDT
>
>A domstats field has been created to show the utilization of RDT resources, the command is like this:
>```
> [root at dl-c200 libvirt]# virsh domstats --resctrl
> Domain: 'vm1'
> resctrl.cmt=0
>
> Domain: 'vm3'
> resctrl.cmt=180224
>
> Domain: 'vm2'
> resctrl.cmt=2613248
>```
>
>
>Wang Huaqiang (3):
> util: add Intel x86 RDT/CMT support
> tools: virsh: add command for controling/monitoring resctrl
> tools: virsh domstats: show RDT CMT resource utilization information
>
> include/libvirt/libvirt-domain.h | 10 ++
> src/conf/domain_conf.c | 28 ++++
> src/conf/domain_conf.h | 3 +
> src/driver-hypervisor.h | 8 +
> src/libvirt-domain.c | 92 +++++++++++
> src/libvirt_private.syms | 9 +
> src/libvirt_public.syms | 6 +
> src/qemu/qemu_driver.c | 189 +++++++++++++++++++++
> src/qemu/qemu_process.c | 65 +++++++-
> src/remote/remote_daemon_dispatch.c | 45 +++++
> src/remote/remote_driver.c | 2 +
> src/remote/remote_protocol.x | 28 +++-
> src/remote_protocol-structs | 12 ++
> src/util/virresctrl.c | 316 +++++++++++++++++++++++++++++++++++-
> src/util/virresctrl.h | 44 +++++
> tools/virsh-domain-monitor.c | 7 +
> tools/virsh-domain.c | 74 +++++++++
> 17 files changed, 933 insertions(+), 5 deletions(-)
>
>--
>2.7.4
>
>--
>libvir-list mailing list
>libvir-list at redhat.com
>https://www.redhat.com/mailman/listinfo/libvir-list
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: Digital signature
URL: <http://listman.redhat.com/archives/libvir-list/attachments/20180611/5b24aef5/attachment-0001.sig>
More information about the libvir-list
mailing list