[libvirt] [RFC PATCH v2 3/3] qemu: memory pre-pinning support for RDMA migration

[Michael R. Hines]

On 02/04/2014 10:56 PM, Jiri Denemark wrote:
> On Mon, Jan 13, 2014 at 14:28:12 +0800, mrhines at linux.vnet.ibm.com wrote:
>> From: "Michael R. Hines" <mrhines at us.ibm.com>
>>
>> RDMA Live migration requires registering memory with the hardware,
> Hmm, I forgot to ask when I was reviewing the previous patch but does
> any of this RDMA migration functionality require special privileges or
> is any unprivileged process able to use RDMA?

No, it does not require any special privileges (just like HPC programs),
but it does access a bunch of special device files (unprivleged):

I believe someone recommended that we had the list of those
device files to the default list of allowed devices that libvirt is
already maintaining.....

I'll include them in the next patch....

> +{
> +    qemuDomainObjPrivatePtr priv = vm->privateData;
> +    int ret;
> +
> +    if (qemuDomainObjEnterMonitorAsync(driver, vm, job) < 0)
> +        return -1;
> +
> +    ret = qemuMonitorGetMigrationCapability(
> +                priv->mon,
> +                QEMU_MONITOR_MIGRATION_CAPS_RDMA_PIN_ALL);
> Is this capability always present when QEMU supports RDMA migration? If
> so, it could be used when we check if QEMU supports RDMA migration.

See my comment earlier in the thread.......

Yes, it's present, but it still does not guarantee that QEMU supports
it if RDMA was compiled out - only the version number is a
(minimal) guarantee, and even then the hardware can still throw
an error if RDMA itself is not supported.

I'm still not seeing what's wrong with depending on the version
number since other features are also depending on the version
number.

Are you guys suggesting that we stop depending on version altogether
for *all* QEMU features?


>> +        unsigned long long memKB = vm->def->mem.hard_limit ?
>> +                                   vm->def->mem.hard_limit :
>> +                vm->def->mem.max_balloon + 1024 * 1024;
>> +                virProcessSetMaxMemLock(vm->pid, memKB * 3);
> Apart from weird indentation of the virProcessSetMaxMemLock line, there
> are several issues here:
>
> - this code should be moved inside qemuMigrationSetPinAll and done only
>    if VIR_MIGRATE_RDMA_PIN_ALL flag was used.
>
> - virProcessSetMaxMemLock wants the value in bytes, thus memKB should
>    rather by multiplied by 1024.
>
> - what memory is going to be mlock()ed with rdma-pin-all? Is it going to
>    be all memory allocated by QEMU or just the domain's memory? If it's
>    all QEMU memory, we already painfully found out it's impossible to
>    automatically compute how much memory QEMU consumes in addition to the
>    configured domain's memory and I think a better approach would be to
>    just migration with rdma-pin-all unless hard_limit is specified.
(Acknowledged for the first two comments).

Regarding your 3rd part: That's why I multiplied the number by 3,
the RDMA code *must* lock or the whole thing falls apart, so
we have to make "some kind" of reasonable assumption on how
much to set the lock limit to.

I'm willing to go even higher than 3 times the limit, if nobody objects,
but we have to pick some kind of limit......somehow.

Comments?

- Michael