[libvirt] question about rdma migration

[Michael R. Hines]

Besides, If it didn't work as root or qemu, then you simply didn't get 
the configuration setup correctly.

I advise you to get it working correctly first (via opening another 
shell and verifying that the limits are set by default)
before embarking on a change to libvirt.

/*
  * Michael R. Hines
  * Platform Engineer, DigitalOcean.
  */

On 02/19/2016 04:37 PM, Roy Shterman wrote:
> Yes,
>
> I tried also running it as root user and it also didn't worked.
>
> Do you know where libvirt (or QEMU) gets the value for process 
> MEMLOCK? maybe i can change this value in libvirt code?
>
> Regards,
> Roy
>
> On Fri, Feb 19, 2016 at 11:15 PM, Michael R. Hines 
> <mhines at digitalocean.com <mailto:mhines at digitalocean.com>> wrote:
>
>     Is the QEMU process (after startup) actually running as the QEMU
>     userid ?
>
>     /*
>       * Michael R. Hines
>       * Platform Engineer, DigitalOcean.
>       */
>
>     On 02/19/2016 02:43 PM, Roy Shterman wrote:
>>     First off all thank you for your answer,
>>
>>     I couldn't figured how to start virtual machine with increased
>>     MEMLOCK,
>>
>>     tried to add into /etc/security/limits.d
>>
>>     qemu            soft    memlock  3221225
>>     qemu            hard    memlock  3221225
>>
>>     so max locked-in-memory will be 3G, but it didn't worked.
>>
>>     still has MEMLOCK of 60kb per each VM.
>>
>>     Maybe you can spot what I'm doing wrong?
>>
>>     On Tue, Feb 9, 2016 at 5:16 PM, Michael R. Hines
>>     <michael at hinespot.com <mailto:michael at hinespot.com>> wrote:
>>
>>         Hi Roy,
>>
>>         On 02/09/2016 03:57 AM, Roy Shterman wrote:
>>
>>             Hi,
>>
>>             I tried to understand the rdma-migration in qemu code and
>>             i have two questions about it:
>>
>>             1. I'm working with qemu-kvm using libvirt and i'm getting
>>
>>             MEMLOCK    max locked-in-memory address space     65536
>>             65536 bytes
>>
>>             in qemu process so I don't understand how can you use
>>             rdma-pin-all with such low MEMLOCK.
>>
>>             I found a solution in libvirt to lock all vm memory in
>>             advance and to enlarge MEMLOCK.
>>             It uses memoryBacking locking and memory tuning
>>             hard_limit of vm memory but I couldn't find a usage of
>>             this in rdma-migration code.
>>
>>
>>         You're absolutey right, the RDMA migration code itself
>>         doesn't set this lock limit explicitly because there are
>>         system-wide restrictions in both appArmour,
>>         /etc/security, as well as SELINUX that restrict applications
>>         from arbitrarily setting their maximum memory lock limits.
>>
>>         The other problem is CGROUPS: If someone sets a cgroup
>>         control for maximum memory and forgets about that mlock()
>>         limits, then
>>         there will be a conflict.
>>
>>         So, libvirt must have a policy to deal with all of these
>>         possibilities, not just handle a special case for RDMA migration.
>>
>>         The only way "simple" way (without patching the problems
>>         above) to apply a higher lock limit to QEMU is to set the
>>         ulimit for libvirt
>>         (or for QEMU if starting QEMU manually) in your environment
>>         or the command line with $ ulimit # before attempting the
>>         migration,
>>         then the RDMA subsystem will be able to lock the memory
>>         successfully.
>>
>>         The other option is to use /etc/security/limits.conf and set
>>         the option for a specific libvirt process user and make sure
>>         your libvirt/qemu
>>         are not running as root.
>>
>>         QEMU itself also has a "mlock" option built into the command
>>         line, but it also suffers from the same problem --- you have
>>         to find
>>         a way (currently) to increase the limit before using the option.
>>
>>             2. Do you have any comparison of IOPS and bandwidth
>>             between TCP migration and rdma migration?
>>
>>         Yes, lots of comparisons.
>>
>>         http://wiki.qemu.org/Features/RDMALiveMigration
>>         http://www.canturkisci.com/ETC/papers/IBMJRD2011/preprint.pdf
>>
>>
>>             Regards,
>>             Roy
>>
>>
>>
>>
>>
>>
>>     --
>>     libvir-list mailing list
>>     libvir-list at redhat.com <mailto:libvir-list at redhat.com>
>>     https://www.redhat.com/mailman/listinfo/libvir-list
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/libvir-list/attachments/20160219/5f1e6060/attachment-0001.htm>