[libvirt] cpu affinity, isolcpus and cgroups
henning.schild at siemens.com
Thu Jul 2 14:42:47 UTC 2015
On Thu, 2 Jul 2015 15:18:46 +0100
"Daniel P. Berrange" <berrange at redhat.com> wrote:
> On Thu, Jul 02, 2015 at 04:02:58PM +0200, Henning Schild wrote:
> > Hi,
> > i am currently looking into realtime VMs using libvirt. My first
> > starting point was reserving a couple of cores using isolcpus and
> > later tuning the affinity to place my vcpus on the reserved pcpus.
> > My first observation was that libvirt ignores isolcpus. Affinity
> > masks of new qemus will default to all cpus and will not be
> > inherited from libvirtd. A comment in the code suggests that this
> > is done on purpose.
> Ignore realtime + isolcpus for a minute. It is not unreasonable for
> the system admin to decide system services should be restricted to
> run on a certain subset of CPUs. If we let VMs inherit the CPU
> pinning on libvirtd, we'd be accidentally confining VMs to a subset
> of CPUs too. With new cgroups layout, libvirtd lives in a separate
> cgroups tree /system.slice, while VMs live in /machine.slice. So
> for both these reasons, when starting VMs, we explicitly ignore
> any affinity libvirtd has and set VMs mask to allow any CPU.
Sure, that was my first guess as well. Still i wanted to raise the
topic again from the realtime POV.
I am using a pretty recent libvirt from git but did not come across the
system.slice yet. Might be a matter of configuration/invocation of
> > After that i changed the code to use only the available cpus by
> > default. But taskset was still showing all 'f's on my qemus. Then i
> > traced my change down to sched_setaffinity assuming that some other
> > mechanism might have reverted my hack, but it is still in place.
> From the libvirt POV, we can't tell whether the admin set isolcpus
> because they want to reserve those CPUs only for VMs, or because
> they want to stop VMs using those CPUs by default. As such libvirt
> does not try to interpret isolcpus at all, it leaves it upto a
> higher level app to decide on this policy.
I know, you have to tell libvirt that the reservation is actually for
libvirt. My idea was to introduce a config option in libvirt and maybe
sanity check it by looking at whether the pcpus are actually reserved.
Rik recently posted a patch to allow easy programmatic checking of
isolcpus via sysfs.
> In the case of OpenStack, the /etc/nova/nova.conf allows a config
> setting 'vcpu_pin_set' to say what set of CPUs VMs should be allowed
> to run on, and nova will then update the libvirt XML when starting
> each guest.
I see, would it not still make sense to have that setting centrally in
libvirt? I am thinking about people not using nova but virsh or
> > Libvirt is setting up cgroups and now my suspicion is that cgroups
> > and taskset might not work well together.
> > > /sys/fs/cgroup/cpu/machine.slice/machine-qemu\x2dvm1.scope/vcpu0#
> > > cpuacct.usage_percpu
> > > 247340587 50851635 89631114 23383025 412639264 1241965 55442753
> > > 19923 14093629 15863859 27403280 1292195745 82031088 53690508
> > > 135826421 124915000 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> > Looks like the last 16 cores are not used.
> > But if i use taskset to ask for the affinity mask i get all 32 cpus.
> > > taskset -p `cat tasks`
> > > pid 12905's current affinity mask: ffffffff
> > I know that is not strictly libvirt but also a kernel question,
> > still you guys are probably able to point me to what i am missing
> > here.
> > > Linux 3.18.11+ #4 SMP PREEMPT RT
> BTW, I dropped Osier from the CC list, since he no longer works
> as Red Hat.
Yeah, the reply from my mailserver suggested that.
More information about the libvir-list