[libvirt] question about rdma migration

[Michael R. Hines]

Hi Roy,

On 02/09/2016 03:57 AM, Roy Shterman wrote:
> Hi,
>
> I tried to understand the rdma-migration in qemu code and i have two 
> questions about it:
>
> 1. I'm working with qemu-kvm using libvirt and i'm getting
>
> MEMLOCK    max locked-in-memory address space     65536 65536 bytes
>
> in qemu process so I don't understand how can you use rdma-pin-all 
> with such low MEMLOCK.
>
> I found a solution in libvirt to lock all vm memory in advance and to 
> enlarge MEMLOCK.
> It uses memoryBacking locking and memory tuning hard_limit of vm 
> memory but I couldn't find a usage of this in rdma-migration code.
>

You're absolutey right, the RDMA migration code itself doesn't set this 
lock limit explicitly because there are system-wide restrictions in both 
appArmour,
/etc/security, as well as SELINUX that restrict applications from 
arbitrarily setting their maximum memory lock limits.

The other problem is CGROUPS: If someone sets a cgroup control for 
maximum memory and forgets about that mlock() limits, then
there will be a conflict.

So, libvirt must have a policy to deal with all of these possibilities, 
not just handle a special case for RDMA migration.

The only way "simple" way (without patching the problems above) to apply 
a higher lock limit to QEMU is to set the ulimit for libvirt
(or for QEMU if starting QEMU manually) in your environment or the 
command line with $ ulimit # before attempting the migration,
then the RDMA subsystem will be able to lock the memory successfully.

The other option is to use /etc/security/limits.conf and set the option 
for a specific libvirt process user and make sure your libvirt/qemu
are not running as root.

QEMU itself also has a "mlock" option built into the command line, but 
it also suffers from the same problem --- you have to find
a way (currently) to increase the limit before using the option.

> 2. Do you have any comparison of IOPS and bandwidth between TCP 
> migration and rdma migration?
>
Yes, lots of comparisons.

http://wiki.qemu.org/Features/RDMALiveMigration
http://www.canturkisci.com/ETC/papers/IBMJRD2011/preprint.pdf


> Regards,
> Roy
>
>