<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<br>
<blockquote type="cite"
cite="mid:2c03449d-16dd-d0d3-ab10-757c20c152bc@gmail.com">
<blockquote type="cite"
cite="mid:CAAhRwWLt8UUyi=nkvR4wJ6A3_RUZ-KDMon9hsJG9wN2FwOQaWg@mail.gmail.com">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex"> <br>
@Eli: Can you help with the testing?<br>
<br>
</blockquote>
</div>
</div>
</blockquote>
</blockquote>
<br>
It seems the interface is only implement the isolated case, I
remember that you have proposed that for some overlap case? <br>
<br>
I have not see the whole patch set yet, but I have some quick
testing on you patch, will try to find more time to review patches
(Currently I am maintain another daemon software which is dedicated
for RDT feature called RMD)<br>
<br>
Only the issue 1 is the true issue, for the others, I think they
should be discussed, or be treat as the 'known issue'.<br>
<br>
My env:<br>
<br>
L1d cache: 32K<br>
L1i cache: 32K<br>
L2 cache: 256K<br>
L3 cache: 56320K<br>
NUMA node0 CPU(s): 0-21,44-65<br>
NUMA node1 CPU(s): 22-43,66-87<br>
<br>
<br>
virsh capabilities:<br>
<br>
171
<cache>
<br>
172 <bank id='0' level='3' type='both' size='55' unit='MiB'
cpus='0-21,44-65'><br>
173 <control granularity='2816' unit='KiB' type='both'
maxAllocs='16'/> <br>
174
</bank>
<br>
175 <bank id='1' level='3' type='both' size='55' unit='MiB'
cpus='22-43,66-87'><br>
176 <control granularity='2816' unit='KiB' type='both'
maxAllocs='16'/> <br>
177
</bank>
<br>
178 </cache> <br>
<br>
<b>Issue:<br>
<br>
</b>1. Doesn't support asynchronous cache allocation. e.g, I need
provide all cache allocation require ways, but I am only care about
the allocation on one of the cache id, cause the VM won't be
schedule to another cache (socket).<br>
<br>
So I got this error if I define the domain like this:<br>
<br>
<vcpu placement='static'>6</vcpu><br>
<cputune><br>
<emulatorpin cpuset='0,37-38,44,81-82'/><br>
<cachetune vcpus='0-4'><br>
<b> <cache id='0' level='3' type='both' size='2816'
unit='KiB'/><br>
^^^ not provide cache id='1'<br>
</b> </cachetune><br>
<br>
<br>
root@s2600wt:~# virsh start kvm-cat<br>
error: Failed to start domain kvm-cat<br>
error: Cannot write into schemata file
'/sys/fs/resctrl/qemu-qemu-13-kvm-cat-0-4/schemata': Invalid
argument<br>
<br>
This behavior is not correct.<br>
<br>
I expect the CBM will be look like:<br>
<br>
root@s2600wt:/sys/fs/resctrl# cat qemu-qemu-14-kvm-cat-0-4/*<br>
000000,00000000,00000000<br>
L3:0=80;1=fffff <b>(no matter what it is, cause my VM won't be
schedule on it, ether I have deinfe the vcpu->cpu pining or, I
assume that kernel won't schedule it to cache 1)<br>
<br>
</b>Or at least, restrict xml when I define this domain, tell me I
need to provide all cache ids (even if I have 4 cache but I only run
my VM on 'cache 0')<br>
<b><br>
</b>2. cache way fragment (no good answers)<br>
<br>
I see that for now we allocate cache ways start from the low bits,
newly created VM will allocate cache from the next way, if some of
the VM (allocated ways in the middle, eg it's schemata is 00100)
destroyed, and that slot (1 cache way) may not fit others and it
will be wasted, But, how can we handle this, seems no good way,
rearrangement? That will lead cache missing in a time window I
think. <br>
<br>
3. The admin/user should manually operate the default resource
group, that's is to say, after resctrl is mounted, the admin/user
should manually change the schemata of default group. Will libvirt
provide interface/API to handle it?<br>
<br>
4. Will provide some APIs like `FreeCacheWay` to end user to see how
many cache ways could be allocated on the host?<br>
<br>
For other users/orchestrator (nova), they may need to know if a
VM can schedule on the host, but the cache ways is not liner, it may
have fragment.<br>
<br>
5, What if other application want to have some shared cache ways
with some of the VM?<br>
Libvirt for now try to read all of the resource group (instead
of maintain the consumed cache ways itself), so if another resource
group was created under /sys/fs/resctl, and the schemata of it is
"FFFFF", then libvirt will report not enough room for new VM. But
the user actually want to have another Appliation(e.g. ovs, dpdk
pmds) share cache ways with the VM created by libvirt.<br>
<b></b><br>
I will try to more cases..<br>
<br>
Thanks Eli.<br>
<br>
<blockquote type="cite"
cite="mid:2c03449d-16dd-d0d3-ab10-757c20c152bc@gmail.com">
<blockquote type="cite"
cite="mid:CAAhRwWLt8UUyi=nkvR4wJ6A3_RUZ-KDMon9hsJG9wN2FwOQaWg@mail.gmail.com">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex"> Martin
Kletzander (9):<br>
Rename virResctrlInfo to virResctrlInfoPerCache<br>
util: Add virResctrlInfo<br>
conf: Use virResctrlInfo in capabilities<br>
util: Remove now-unneeded resctrl functions<br>
resctrl: Add functions to work with resctrl allocations<br>
conf: Add support for cputune/cachetune<br>
tests: Add virresctrltest<br>
qemu: Add support for resctrl<br>
docs: Add CAT (resctrl) support into news.xml<br>
<br>
docs/<a href="http://formatdomain.html.in"
rel="noreferrer" target="_blank" moz-do-not-send="true">formatdomain.html.in</a>
| 54 +<br>
docs/news.xml | 9
+<br>
docs/schemas/domaincommon.rng | 32
+<br>
po/POTFILES.in | 1
+<br>
src/Makefile.am | 2
+-<br>
src/conf/capabilities.c | 55
+-<br>
src/conf/capabilities.h | 4
+-<br>
src/conf/domain_conf.c | 251
++++<br>
src/conf/domain_conf.h | 13
+<br>
src/libvirt_private.syms | 16
+-<br>
src/qemu/qemu_process.c | 61
+-<br>
src/util/virresctrl.c | 1380
++++++++++++++++++--<br>
src/util/virresctrl.h | 86
+-<br>
src/util/virresctrlpriv.h | 27
+<br>
tests/Makefile.am | 8
+-<br>
tests/genericxml2xmlindata/<wbr>cachetune-cdp.xml
| 36 +<br>
.../cachetune-colliding-<wbr>allocs.xml
| 30 +<br>
.../cachetune-colliding-tunes.<wbr>xml
| 32 +<br>
.../cachetune-colliding-types.<wbr>xml
| 30 +<br>
tests/genericxml2xmlindata/<wbr>cachetune-small.xml
| 29 +<br>
tests/genericxml2xmlindata/<wbr>cachetune.xml
| 33 +<br>
tests/genericxml2xmltest.c | 10
+<br>
tests/virresctrldata/resctrl-<wbr>cdp.schemata
| 2 +<br>
.../virresctrldata/resctrl-<wbr>skx-twocaches.schemata
| 1 +<br>
tests/virresctrldata/resctrl-<wbr>skx.schemata
| 1 +<br>
tests/virresctrldata/resctrl.<wbr>schemata
| 1 +<br>
tests/virresctrltest.c | 102
++<br>
27 files changed, 2174 insertions(+), 132 deletions(-)<br>
create mode 100644 src/util/virresctrlpriv.h<br>
create mode 100644 tests/genericxml2xmlindata/<wbr>cachetune-cdp.xml<br>
create mode 100644 tests/genericxml2xmlindata/<wbr>cachetune-colliding-allocs.xml<br>
create mode 100644 tests/genericxml2xmlindata/<wbr>cachetune-colliding-tunes.xml<br>
create mode 100644 tests/genericxml2xmlindata/<wbr>cachetune-colliding-types.xml<br>
create mode 100644 tests/genericxml2xmlindata/<wbr>cachetune-small.xml<br>
create mode 100644 tests/genericxml2xmlindata/<wbr>cachetune.xml<br>
create mode 100644 tests/virresctrldata/resctrl-<wbr>cdp.schemata<br>
create mode 100644 tests/virresctrldata/resctrl-<wbr>skx-twocaches.schemata<br>
create mode 100644 tests/virresctrldata/resctrl-<wbr>skx.schemata<br>
create mode 100644 tests/virresctrldata/resctrl.<wbr>schemata<br>
create mode 100644 tests/virresctrltest.c<br>
<span class="HOEnZb"><font color="#888888"><br>
--<br>
2.15.1<br>
<br>
</font></span></blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</blockquote>
<br>
</body>
</html>