[libvirt] [v4 0/9] Support cache tune in libvirt

[Marcelo Tosatti]

On Tue, Feb 14, 2017 at 04:59:52PM +0800, Eli Qiao wrote:
> 
> 
> --  
> Eli Qiao
> Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
> 
> 
> On Tuesday, 14 February 2017 at 4:37 PM, Martin Kletzander wrote:
> 
> > On Mon, Feb 13, 2017 at 04:09:13PM -0200, Marcelo Tosatti wrote:
> > > On Mon, Feb 13, 2017 at 03:42:04PM +0800, Eli Qiao wrote:
> > > > > > L3data:0=0xff00;...
> > > > > > L3code:0=0xff00;...
> > > > > >  
> > > > > > * Would you please help to test if the functions work.
> > > > >  
> > > > > Setting up CDP machine.
> > > > >  
> > > > > Unrelated:
> > > > >  
> > > > > Found a bug:
> > > > >  
> > > > > The code should scan for all directories in resctrlfs,
> > > > > and then find free CBM space from that:
> > > > >  
> > > > >  
> > > > > free_cbm_space = ~(resctrldir1.CBM_bits &
> > > > > resctrldir2.CBM_bits &
> > > > > ...
> > > > > resctrldirN.CBM_bits)
> > > > >  
> > > > > For all resctrlfs directories.
> > > > >  
> > > > > The bug is as follows:
> > > > >  
> > > > > Create a directory in resctrlfs by hand:
> > > > >  
> > > > > # mkdir newres
> > > >  
> > > > Libvirt will not aware this after it starts running, so we should not allow operate /sys/fs/.
> > >  
> > > Are you saying that usage of the resctrlfs filesystem should not be
> > > allowed after libvirt starts? I don't think this is correct.
> > >  
> >  
> >  
> > In some cases the only way to accomplish keeping consistency would be
> > saying that if you are using libvirt, you should not change anything
> > behind its back. It would be really wrong in this case, mainly when you
> > need to set the allocation for other apps, especially when there is
> > nicely designed file that you can lock.
> >  
> 
> That will mess up the result if allow other app to operate /sys/fs/resctrl, we consume cache and then change the default schemata, if other apps
> create a schemata which has overlap with our existed one, the allocation will fail. which is saying the allocated cache will be shared between VMs and other apps.

This is why the other app and libvirt should use the filesystem lock. 

> My concern is that if some app create a new schemata, by default it’s fffff, in our calculate process, the default schemata will be 00001 (kernel don’t allow us for 0), which indicate no cache left …
> libvirtd will fail to boot VM.

If both libvirtd and all apps use the filesystem lock, and do not use
CBM space from other CAT reservations, then there will be no problem.

> I really don’t want to let user and libvirtd operate /sys/fs/resctrl at same time.

This is necessary for the NFV usecase. Example: 

OVS-DPDK receiving packets from network card 
and forwarding them to VMs.

Both OVS-DPDK and the VMs use the resctrl filesystem.


> > > > we will scan for all directors while the libvirt daemon begin running, and libvirt will remove exist directories if no tasks inside of it.
> > >  
> >  
> >  
> > Definitely not. What if someone wants to create another allocation?
> > They start by creating a directory, then libvirtd removes it before they
> > manage to add anything to tasks.
> >  
> >  
> 
> libvirtd try to scan them which has no tasks to keep things cleanup, think about that some VMs may fail due to unexpected reason, there task ids are lost and kernel remove them from resctrl.
> this causes cache leak.
>   
> >  
> > >  
> > >  
> > > >  
> > > >  
> > > > > # cd newres
> > > > > # echo "L3:0=3;1=f0000" > schemata
> > > > > # virsh start guest
> > > > > # cd ../b4c270b5-e0f9-4106-a446-69032872ed7e
> > > > > # cat schemata
> > > > > L3:0=3;1=f0000
> > > > >  
> > > > > That is, it is using the same CBM space as the "newres"
> > > > > reservation.
> > > > >  
> > > >  
> > > >  
> > > >  
> > > > As user create a new directory after libvirt running, it don’t notice newly created directory under /sys/fs/resctrl.
> > > >  
> > > > That will lead mess, this should be forbidden.
> > >  
> > > Well, this means that only libvirt can use the resctrlfs filesystem.
> > >  
> > > This forbids other applications which require allocation
> > > of CAT resources from being used.
> > >  
> > > Its simple to fix it: move the scanning of resctrlfs data from libvirt
> > > initialization time to:
> > >  
> > > - VM initialization time
> > > and
> > > - virConnectGetCapabilities time
> > >  
> > > The second case is necessary to get updated free space information.
> >  
> > Just VM initialization time could be enough as virConnectGetCapabilities
> > would just know the total and free size would be reported in an API (if
> > I rememer the discussion correctly)
> >  
> >  
> 
> 
> By doing this, it will slowdown the process of booting a VM if we scan it every time...  

This is OK, if performance turns out to be a problem the code can 
be optimized.

But first it has to be implemented correctly, which means supporting
other users of resctrl filesystem (see the OVS-DPDK example above).

> >  
> > Martin
> >  
>