[libvirt] Automatically affinitize hostdev interrupts to vm vCpus (qemu/kvm)

[Mooney, Sean K]

Hi sorry for the delay in responding.

Regarding the intel email footer,  I understand that It might deter people from responding.
it is automatically added if I email an external address by our exchange server.
I have asked for my account to be added to the exception whitelist 
so this should be removed soon. If not I will follow up with my personal  email account.

Regarding abstracting the implementation to a separate file, if the changes are small enough 
Perhaps the new code can be added util/virpci and util/virhostdev instead of creating util/virinterupt  
I can refactor the implementation to reflect whichever is preferred during code review.

I would prefer to use auto as the default also.  
If  the consensus is that it would be better not to enable this feature by default, then I am happy to  do that too.  
>From my initial investigation I have just looked at affinitizing interrupts when the vm starts.
I agree that the  final implementation should also handle changes in the vm configuration (vpus, hostdevs plug/unplugged) 
and changes In the vm power state also.

Regards
Sean


-----Original Message-----
From: Martin Kletzander [mailto:mkletzan at redhat.com] 
Sent: Monday, August 25, 2014 11:35 AM
To: Mooney, Sean K
Cc: libvir-list at redhat.com
Subject: Re: [libvirt] Automatically affinitize hostdev interrupts to vm vCpus (qemu/kvm)

On Mon, Aug 18, 2014 at 07:01:40PM +0000, Mooney, Sean K wrote:
>Hi
>I would like to ask for comments and propose a possible new libvirt feature.
>

Hi, could you convince your e-mail agent to wrap long lines?

>Problem statement:
>At present when you boot a vm via libvirt it is possible  to pin vm vCPUs to host CPUs to improve performance in the  guest under certain conditions.
>If hostdev interrupts are not pined to a specific core/cpuset suboptimal processing of irqs may occur reducing the performance of both guest and host.
>I would like to propose extending libvirt to automatically pin interrupts for hostdev devices if they are present in the guest.
>
>By affinitizing interrupts, cache line sharing between the specified interrupt and guest can be achieved.
>If CPU affinity for the guest, as set by the cpuset parameter, is 
>intelligently chosen to place the guest on the same numa node as the hostdev, cross socket traffic can be mitigated.
>As a result, latency which would be introduce if the interrupt processing was scheduled to a non-local numa node CPU can be reduced via interrupt pinning.
>
>Proposed change:
>
>*         util/virpci and util/virhostdev will be extended to retrieve IRQ and msi_interupt information from sysfs.
>
>*         util/virinterupt  will be created
>
>*         util/virinterupt  will implement managing interrupt affinity via /proc/irq/x/smp_affinity
>

Nice that this will be abstracted out in a separate file, although if it's not huge, it can be part of something else.

>*         qemuProcessInitCpuAffinity will be extended to conditionally affinitize hostdev interrupts to vm vCpus when hostdevs are present in the vm definition.

So it will only affect hostdevs, OK, that means there should be less (or no) disadvantages).  Although beware that hostdevs can be plugged/unplugged, number of vCPUs can be changed and, most importantly, the affinities (either set with sched_set_affinity or using cgroups) can be changed during the lifetime of the VM, and the smp_affinity of each IRQ should track such changes.  Needless to say the affinity should be restored after the machine dies/is turned off, etc., not just on hostdev unplug.

>Alternative implementation:
>
>In addition to the above changes the hostdev element could be extended to include a cpuset attribute:
>
>*         if the cpuset is auto: the interrupts would be pinned to the same cpuset as the vms vCPU
>
>*         if a cpuset is specified: the interrupts would be affinitized as per the set cpuset
>
>*         if the cpuset is native: the interrupts will not be pinned and the current behaviour will not be changed.
>
>*         If the cpuset attribute is not present either the behaviour of auto or native could be used as a default.
>
>o   Using auto as the default would allow transparent use of the feature.
>

I like defaulting to auto also because the transparent use will have effect only in existing deployments with vCPU pinning.

>o   Using native as the default would allow no changes to any existing deployments unless the feature is requested.

Although this version might be preferred by others.  This can, however, be discussed (and changed) during reviews.

>Any feedback is welcomed.
>Regards
>Sean.
>--------------------------------------------------------------
>Intel Shannon Limited
>Registered in Ireland
>Registered Office: Collinstown Industrial Park, Leixlip, County Kildare 
>Registered Number: 308263 Business address: Dromore House, East Park, 
>Shannon, Co. Clare
>
>This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
>

Since this mailing-list is public and archived, such disclaimer is unenforceable.  As an upstream project, the development is done in the open.  Such statements as the above one may be the cause of nobody replying for some time to your e-mail.

The statement itself can be understood that even delivering the mail through a list is prohibited.  Please consider removing such statements in following e-mails (or use private e-mail address if the disclaimer is enforced by your employer) as keeping them may result in the rest of the community not being able to communicate with you.

Martin
--------------------------------------------------------------
Intel Shannon Limited
Registered in Ireland
Registered Office: Collinstown Industrial Park, Leixlip, County Kildare
Registered Number: 308263
Business address: Dromore House, East Park, Shannon, Co. Clare

This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.