<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
Is the QEMU process (after startup) actually running as the QEMU
userid ?<br>
<pre class="moz-signature" cols="72">/*
* Michael R. Hines
* Platform Engineer, DigitalOcean.
*/</pre>
<div class="moz-cite-prefix">On 02/19/2016 02:43 PM, Roy Shterman
wrote:<br>
</div>
<blockquote
cite="mid:CAOBqJ55OTaviJeVoJcjPN0zcXbi=+q24qbxtSr2=pn-OWkZqFQ@mail.gmail.com"
type="cite">
<div dir="ltr">First off all thank you for your answer,
<div><br>
</div>
<div>I couldn't figured how to start virtual machine with
increased MEMLOCK,</div>
<div><br>
</div>
<div>tried to add into /etc/security/limits.d </div>
<div><br>
</div>
<div>
<div>qemu soft memlock 3221225</div>
<div>qemu hard memlock 3221225</div>
</div>
<div><br>
</div>
<div>so max locked-in-memory will be 3G, but it didn't worked.</div>
<div><br>
</div>
<div>still has MEMLOCK of 60kb per each VM.</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">Maybe you can spot what I'm doing
wrong?</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Feb 9, 2016 at 5:16 PM,
Michael R. Hines <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:michael@hinespot.com" target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:michael@hinespot.com">michael@hinespot.com</a></a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Roy,<span
class=""><br>
<br>
On 02/09/2016 03:57 AM, Roy Shterman wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi,<br>
<br>
I tried to understand the rdma-migration in qemu code
and i have two questions about it:<br>
<br>
1. I'm working with qemu-kvm using libvirt and i'm
getting<br>
<br>
MEMLOCK max locked-in-memory address space
65536 65536 bytes<br>
<br>
in qemu process so I don't understand how can you use
rdma-pin-all with such low MEMLOCK.<br>
<br>
I found a solution in libvirt to lock all vm memory in
advance and to enlarge MEMLOCK.<br>
It uses memoryBacking locking and memory tuning
hard_limit of vm memory but I couldn't find a usage of
this in rdma-migration code.<br>
<br>
</blockquote>
<br>
</span>
You're absolutey right, the RDMA migration code itself
doesn't set this lock limit explicitly because there are
system-wide restrictions in both appArmour,<br>
/etc/security, as well as SELINUX that restrict
applications from arbitrarily setting their maximum memory
lock limits.<br>
<br>
The other problem is CGROUPS: If someone sets a cgroup
control for maximum memory and forgets about that mlock()
limits, then<br>
there will be a conflict.<br>
<br>
So, libvirt must have a policy to deal with all of these
possibilities, not just handle a special case for RDMA
migration.<br>
<br>
The only way "simple" way (without patching the problems
above) to apply a higher lock limit to QEMU is to set the
ulimit for libvirt<br>
(or for QEMU if starting QEMU manually) in your
environment or the command line with $ ulimit # before
attempting the migration,<br>
then the RDMA subsystem will be able to lock the memory
successfully.<br>
<br>
The other option is to use /etc/security/limits.conf and
set the option for a specific libvirt process user and
make sure your libvirt/qemu<br>
are not running as root.<br>
<br>
QEMU itself also has a "mlock" option built into the
command line, but it also suffers from the same problem
--- you have to find<br>
a way (currently) to increase the limit before using the
option.<span class=""><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
2. Do you have any comparison of IOPS and bandwidth
between TCP migration and rdma migration?<br>
<br>
</blockquote>
</span>
Yes, lots of comparisons.<br>
<br>
<a moz-do-not-send="true"
href="http://wiki.qemu.org/Features/RDMALiveMigration"
rel="noreferrer" target="_blank">http://wiki.qemu.org/Features/RDMALiveMigration</a><br>
<a moz-do-not-send="true"
href="http://www.canturkisci.com/ETC/papers/IBMJRD2011/preprint.pdf"
rel="noreferrer" target="_blank">http://www.canturkisci.com/ETC/papers/IBMJRD2011/preprint.pdf</a><br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
Regards,<br>
Roy<br>
<br>
<br>
</blockquote>
<br>
</blockquote>
</div>
<br>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">--
libvir-list mailing list
<a class="moz-txt-link-abbreviated" href="mailto:libvir-list@redhat.com">libvir-list@redhat.com</a>
<a class="moz-txt-link-freetext" href="https://www.redhat.com/mailman/listinfo/libvir-list">https://www.redhat.com/mailman/listinfo/libvir-list</a></pre>
</blockquote>
<br>
</body>
</html>