[libvirt] Automatically affinitize hostdev interrupts to vm vCpus (qemu/kvm)

[Martin Kletzander]

On Mon, Aug 18, 2014 at 07:01:40PM +0000, Mooney, Sean K wrote:
>Hi
>I would like to ask for comments and propose a possible new libvirt feature.
>

Hi, could you convince your e-mail agent to wrap long lines?

>Problem statement:
>At present when you boot a vm via libvirt it is possible  to pin vm vCPUs to host CPUs to improve performance in the  guest under certain conditions.
>If hostdev interrupts are not pined to a specific core/cpuset suboptimal processing of irqs may occur reducing the performance of both guest and host.
>I would like to propose extending libvirt to automatically pin interrupts for hostdev devices if they are present in the guest.
>
>By affinitizing interrupts, cache line sharing between the specified interrupt and guest can be achieved.
>If CPU affinity for the guest, as set by the cpuset parameter,
>is intelligently chosen to place the guest on the same numa node as the hostdev, cross socket traffic can be mitigated.
>As a result, latency which would be introduce if the interrupt processing was scheduled to a non-local numa node CPU can be reduced via interrupt pinning.
>
>Proposed change:
>
>*         util/virpci and util/virhostdev will be extended to retrieve IRQ and msi_interupt information from sysfs.
>
>*         util/virinterupt  will be created
>
>*         util/virinterupt  will implement managing interrupt affinity via /proc/irq/x/smp_affinity
>

Nice that this will be abstracted out in a separate file, although if
it's not huge, it can be part of something else.

>*         qemuProcessInitCpuAffinity will be extended to conditionally affinitize hostdev interrupts to vm vCpus when hostdevs are present in the vm definition.

So it will only affect hostdevs, OK, that means there should be less
(or no) disadvantages).  Although beware that hostdevs can be
plugged/unplugged, number of vCPUs can be changed and, most
importantly, the affinities (either set with sched_set_affinity or
using cgroups) can be changed during the lifetime of the VM, and the
smp_affinity of each IRQ should track such changes.  Needless to say
the affinity should be restored after the machine dies/is turned off,
etc., not just on hostdev unplug.

>Alternative implementation:
>
>In addition to the above changes the hostdev element could be extended to include a cpuset attribute:
>
>*         if the cpuset is auto: the interrupts would be pinned to the same cpuset as the vms vCPU
>
>*         if a cpuset is specified: the interrupts would be affinitized as per the set cpuset
>
>*         if the cpuset is native: the interrupts will not be pinned and the current behaviour will not be changed.
>
>*         If the cpuset attribute is not present either the behaviour of auto or native could be used as a default.
>
>o   Using auto as the default would allow transparent use of the feature.
>

I like defaulting to auto also because the transparent use will have
effect only in existing deployments with vCPU pinning.

>o   Using native as the default would allow no changes to any existing deployments unless the feature is requested.

Although this version might be preferred by others.  This can,
however, be discussed (and changed) during reviews.

>Any feedback is welcomed.
>Regards
>Sean.
>--------------------------------------------------------------
>Intel Shannon Limited
>Registered in Ireland
>Registered Office: Collinstown Industrial Park, Leixlip, County Kildare
>Registered Number: 308263
>Business address: Dromore House, East Park, Shannon, Co. Clare
>
>This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
>

Since this mailing-list is public and archived, such disclaimer is
unenforceable.  As an upstream project, the development is done in the
open.  Such statements as the above one may be the cause of nobody
replying for some time to your e-mail.

The statement itself can be understood that even delivering the mail
through a list is prohibited.  Please consider removing such
statements in following e-mails (or use private e-mail address if the
disclaimer is enforced by your employer) as keeping them may result in
the rest of the community not being able to communicate with you.

Martin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://listman.redhat.com/archives/libvir-list/attachments/20140825/2032f7da/attachment-0001.sig>