[libvirt] [PATCH] qemu: Pin the emulator when only cpuset is specified

[Martin Kletzander]

On 10/19/2012 05:45 AM, Osier Yang wrote:
> On 2012年10月17日 23:38, Martin Kletzander wrote:
>> According to our recent changes (clarifications), we should be pinning
>> qemu's emulator processes using the<vcpu>  'cpuset' attribute in case
>> there is no<emulatorpin>  specified.  This however doesn't work
>> entirely as expected and this patch should resolve all the remaining
>> issues.
>> ---
>>   src/qemu/qemu_cgroup.c  | 25 ++++++++++++++++---------
>>   src/qemu/qemu_cgroup.h  |  4 ++--
>>   src/qemu/qemu_driver.c  |  3 ++-
>>   src/qemu/qemu_process.c | 16 ++++++++--------
>>   4 files changed, 28 insertions(+), 20 deletions(-)
>>

[...]

>> diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c
>> index a94e9c4..e08ec67 100644
>> --- a/src/qemu/qemu_process.c
>> +++ b/src/qemu/qemu_process.c
>> @@ -2024,11 +2024,12 @@ cleanup:
>>       return ret;
>>   }
>>
>> -/* Set CPU affinities for emulator threads if emulatorpin xml
>> provided. */
>> +/* Set CPU affinities for emulator threads. */
>>   static int
>>   qemuProcessSetEmulatorAffinites(virConnectPtr conn,
>>                                   virDomainObjPtr vm)
>>   {
>> +    virBitmapPtr cpumask;
>>       virDomainDefPtr def = vm->def;
>>       virNodeInfo nodeinfo;
>>       int ret = -1;
>> @@ -2036,15 +2037,14 @@ qemuProcessSetEmulatorAffinites(virConnectPtr
>> conn,
>>       if (virNodeGetInfo(conn,&nodeinfo) != 0)
>>           return -1;
>>
>> -    if (!def->cputune.emulatorpin)
>> -        return 0;
>> -
>> -    if (virProcessInfoSetAffinity(vm->pid,
>> -                                 
>> def->cputune.emulatorpin->cpumask)<  0) {
>> +    if (def->cputune.emulatorpin)
>> +        cpumask = def->cputune.emulatorpin->cpumask;
>> +    else if (def->cpumask)
>> +        cpumask = def->cpumask;
>> +    else
>>           goto cleanup;
>> -    }
>>
>> -    ret = 0;
>> +    ret = virProcessInfoSetAffinity(vm->pid, cpumask);
>>   cleanup:
>>       return ret;
>>   }
> 
> This patch actually duplicates the affinity setting on domain
> process with sched_setaffinity.
> 
> Assume "cpuset" of <vcpu> is specified, and no "<emulatorpin>".
> 
> qemuProcessInitCpuAffinity is excuted between the fork
> and exec, with this patch, qemuProcessSetEmulatorAffinites
> will set the affinity again using sched_setaffinity.
> 
> Cgroup setting is fine though, as there is no previous setting
> with cgroup.
> 
> I'm wondering why the domain process is not pinned to
> pCPUs specified "cpuset" of "<vcpu>" by qemuProcessInitCpuAffinity,
> when <emulatorpin> is not specified.
> 
> Regards,
> Osier

I'm glad you pointed it out.  To be honest, it took me quite a while to
see where this should be set to fix the problem with as few lines as
possible.  As Peter says, there is a mess with the pinning, cgroups and
affinities.  Rewriting of these things (using affinity even though
crgoups are working on the system, etc.) is in my mind, but not right now.

Martin