[libvirt] question about rdma migration

[Roy Shterman]

First off all thank you for your answer,

I couldn't figured how to start virtual machine with increased MEMLOCK,

tried to add into /etc/security/limits.d

qemu            soft    memlock  3221225
qemu            hard    memlock  3221225

so max locked-in-memory will be 3G, but it didn't worked.

still has MEMLOCK of 60kb per each VM.

Maybe you can spot what I'm doing wrong?

On Tue, Feb 9, 2016 at 5:16 PM, Michael R. Hines <michael at hinespot.com>
wrote:

> Hi Roy,
>
> On 02/09/2016 03:57 AM, Roy Shterman wrote:
>
>> Hi,
>>
>> I tried to understand the rdma-migration in qemu code and i have two
>> questions about it:
>>
>> 1. I'm working with qemu-kvm using libvirt and i'm getting
>>
>> MEMLOCK    max locked-in-memory address space     65536 65536 bytes
>>
>> in qemu process so I don't understand how can you use rdma-pin-all with
>> such low MEMLOCK.
>>
>> I found a solution in libvirt to lock all vm memory in advance and to
>> enlarge MEMLOCK.
>> It uses memoryBacking locking and memory tuning hard_limit of vm memory
>> but I couldn't find a usage of this in rdma-migration code.
>>
>>
> You're absolutey right, the RDMA migration code itself doesn't set this
> lock limit explicitly because there are system-wide restrictions in both
> appArmour,
> /etc/security, as well as SELINUX that restrict applications from
> arbitrarily setting their maximum memory lock limits.
>
> The other problem is CGROUPS: If someone sets a cgroup control for maximum
> memory and forgets about that mlock() limits, then
> there will be a conflict.
>
> So, libvirt must have a policy to deal with all of these possibilities,
> not just handle a special case for RDMA migration.
>
> The only way "simple" way (without patching the problems above) to apply a
> higher lock limit to QEMU is to set the ulimit for libvirt
> (or for QEMU if starting QEMU manually) in your environment or the command
> line with $ ulimit # before attempting the migration,
> then the RDMA subsystem will be able to lock the memory successfully.
>
> The other option is to use /etc/security/limits.conf and set the option
> for a specific libvirt process user and make sure your libvirt/qemu
> are not running as root.
>
> QEMU itself also has a "mlock" option built into the command line, but it
> also suffers from the same problem --- you have to find
> a way (currently) to increase the limit before using the option.
>
> 2. Do you have any comparison of IOPS and bandwidth between TCP migration
>> and rdma migration?
>>
>> Yes, lots of comparisons.
>
> http://wiki.qemu.org/Features/RDMALiveMigration
> http://www.canturkisci.com/ETC/papers/IBMJRD2011/preprint.pdf
>
>
> Regards,
>> Roy
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/libvir-list/attachments/20160219/a82be138/attachment-0001.htm>