<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
Hi,
<div><br>
</div>
<div>my problem can be described simply: libvirt can't handle starting dozens of VMs at the same time.</div>
<div><br>
</div>
<div>(technically, it can, but it's really slow.)</div>
<div><br>
</div>
<div>We have an AMD machine with 256 logical cores and 1.5T ram.</div>
<div>On that machine there is roughly 200 VMs.</div>
<div>Each VM is the same: 8GB of RAM, 4 VCPUs. Half of them is Win7 x86, the other half is Win7 x64.</div>
<div>VMs are using qcow2 as the disk image. These images reside in the ramdisk (tmpfs).</div>
<div><br>
</div>
<div>We use these machines for automatic malware analysis, so our scenario consists of this cycle:</div>
<div>- reverting VM to a running state</div>
<div>- execute sample inside of the VM for ~1-2 minutes</div>
<div>- shutdown the VM</div>
<div><br>
</div>
<div>Of course, this results in multiple VMs trying to start at the same time.</div>
<div>At first, reverts/starts are really fast - second or two.</div>
<div>After about a minute, the "revertToSnapshot" suddenly takes 10-15 seconds, which is really unacceptable.</div>
<div>For comparison, we're running the same scenarion on Proxmox, where the revertToSnapshot usually takes 2 seconds.</div>
<div><br>
</div>
<div>Few notes:</div>
<div>- Because of this fast cycle (~2-3 minutes) and because of VMs taking 10-15 seconds to start, there is barely more than 25-30 VMs running at once.</div>
<div> We would really love to utilise the whole potential of such beast machine of ours, and have at least ~100 VMs running at any given time.</div>
<div>- During the time running, the avg. CPU load isn't higher than 25%. Also, there's only about 280 GB of RAM used. Therefore, it's not limitation of our resources.</div>
<div>- When the framwork is running and libvirt is making its best to start our VMs, I noticed that every libvirt operation is suddenly very slow.</div>
<div> Even simple "virsh list [--all]" takes few seconds to complete, even though it finishes instantly when no VM is running/starting.</div>
<div><br>
</div>
<div>I was trying to search for this issue, but didn't really find anything besides this presentation:</div>
<div>https://events19.linuxfoundation.org/wp-content/uploads/2017/12/Scalability-and-Stability-of-libvirt-Experiences-with-Very-Large-Hosts-Marc-Hartmayer-IBM-1.pdf</div>
<div><br>
</div>
<div>However, I couldn't find those commits in your upstream.</div>
<div><br>
</div>
<div>Is this a known issue? Or is there some setting I don't know of which would magically make the VMs start faster?</div>
<div><br>
</div>
<div>As for steps to reproduce - I don't think there is anything special needed. Just try to start/destroy several VMs in a loop.</div>
<div>There is even provided one-liner for that in the presentation above.</div>
<div><br>
</div>
<div>```</div>
<div># For multiple domains:</div>
<div># while virsh start $vm && virsh destroy $vm; do : ; done</div>
<div># → ~30s hang ups of the libvirtd main loop</div>
<div>```</div>
<div><br>
</div>
<div>Best Regards,</div>
<div>Petr</div>
<br>
</div>
</body>
</html>