[libvirt] OpenStack/libvirt CAT interface

Wed Jan 11 10:18:11 UTC 2017

On Tue, Jan 10, 2017 at 02:18:43PM -0200, Marcelo Tosatti wrote:
> 
> There have been queries about the OpenStack interface 
> for CAT:

FYI, there's another mail discussing libvirt design here:

  https://www.redhat.com/archives/libvir-list/2017-January/msg00354.html

> http://bugzilla.redhat.com/show_bug.cgi?id=1299678
> 
> Comment 2 says:
> Sahid Ferdjaoui 2016-01-19 10:58:48 EST
> A spec will have to be addressed, after a first look this feature needs
> some work in several components of Nova to maintain/schedule/consume
> host's cache. I can work on that spec and implementation it when libvirt
> will provides information about cache and feature to use it for guests.
> 
> I could add a comment about parameters to resctrltool, but since
> this depends on the libvirt interface, it would be good to know
> what the libvirt interface exposes first.
> 
> I believe it should be essentially similar to OpenStack's
> "reserved_host_memory_mb":
> 
>         Set the reserved_host_memory_mb to reserve RAM for host
> processes. For
>         the purposes of testing I am going to use the default of 512 MB:
>         reserved_host_memory_mb=512
> 
> But rather use:
> 
>         rdt_cat_cache_reservation=type=code/data/both,size=10mb,cacheid=2;
>                                   type=code/data/both,size=2mb,cacheid=1;...
> 
> (per-vcpu).
> 
> Where cache-id is optional.
> 
> What is cache-id (from Documentation/x86/intel_rdt_ui.txt on recent
> kernel sources):
> Cache IDs
> ---------
> On current generation systems there is one L3 cache per socket and L2
> caches are generally just shared by the hyperthreads on a core, but this
> isn't an architectural requirement. We could have multiple separate L3
> caches on a socket, multiple cores could share an L2 cache. So instead
> of using "socket" or "core" to define the set of logical cpus sharing
> a resource we use a "Cache ID". At a given cache level this will be a
> unique number across the whole system (but it isn't guaranteed to be a
> contiguous sequence, there may be gaps).  To find the ID for each
> logical
> CPU look in /sys/devices/system/cpu/cpu*/cache/index*/id

So it seems like cache ID is something we need to add to the XML
I proposed at

  https://www.redhat.com/archives/libvir-list/2017-January/msg00489.html

> 
> 
> WHAT THE USER NEEDS TO SPECIFY FOR VIRTUALIZATION (KVM-RT)
> ==========================================================
> 
> For virtualization the following scenario is desired,
> on a given socket:
> 
>         * VM-A with VCPUs VM-A.vcpu-1, VM-A.vcpu-2.
>         * VM-B with VCPUs VM-B.vcpu-1, VM-B.vcpu-2.
> 
> With one realtime workload on each vcpu-2.
> 
> Assume VM-A.vcpu-2 on pcpu 3.
> Assume VM-B.vcpu-2 on pcpu 5.
> 
> Assume pcpus 0-5 on cacheid 0.
> 
> We want VM-A.vcpu-2 to have a certain region of cache reserved,
> and VM-B.vcpu-2 as well. vcpu-1 for both VMs can use the default group
> (that is not have reserved L3 cache).
> 
> This translates to the following resctrltool-style reservations:
> 
>         res.vm-a.vcpu-2
> 
>                 type=both,size=VM-A-RESSIZE,cache-id=0
> 
>         res.vm-b.vcpu-2
> 
>                 type=both,size=VM-B-RESSIZE,cache-id=0
> 
> Which translate to the following in resctrlfs:
> 
>         res.vm-a.vcpu-2
> 
>                 type=both,size=VM-A-RESSIZE,cache-id=0
>                 type=both,size=default-size,cache-id=1
>                 ...
> 
>         res.vm-b.vcpu-2
> 
>                 type=both,size=VM-B-RESSIZE,cache-id=0
>                 type=both,size=default-size,cache-id=1
>                 ...
> 
> Which is what we want, since the VCPUs are pinned.
> 
> 
> res.vm-a.vcpu-1 and res.vm-b.vcpu-1 don't need to
> be assigned to any reservation, which means they'll
> remain on the default group.

You've showing type=both here which IIUC, means data
and instruction cache. Is that configuring one cache
that serves both purposes ?  Do we need to be able
to configure them independantly.

> 
> RESTRICTIONS TO THE SYNTAX ABOVE
> ================================
> 
> Rules for the parameters:
> * type=code must be paired with type=data entry.

What does this mean exactly when configuring guests ? Do
we have to configure data + instruction cache on the same
cache ID, do they have to be the same size, or are they
completely independant ?

> ABOUT THE LIST INTERFACE
> ========================
> 
> About an interface for listing the reservations
> of the system to OpenStack.
> 
> I think that what OpenStack needs is to check, before
> starting a guest on a given host, that there is sufficient
> space available for the reservation.
> 
> To do that, it can:
> 
>         1) resctrltool list (the end of the output mentions
>            how much free space available there is), or
>            via resctrlfs directly (have to lock the filesystem,
>            read each directory, AND each schemata, and count
>            number of zero bits).
>         2) Via libvirt
> 
> Should fix resctrltool/API to list amount of contiguous free space

OpenStack, should just use libvirt APIs exclusively - there should not
be any need for it to use other tools if we've designed the libvirt API
correctly.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|