<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Hi,<div class=""><br class=""></div><div class="">I'm facing a strange issue while doing a migration from an hypervisor to another one. The migration takes for ever to start moving the memory.</div><div class="">The VM had no workload what so ever, just a basic ubuntu image. The versions on the hypervisors are: libvirt 1.2.21, qemu 1.2.3</div><div class=""><br class=""></div><div class="">Command to launche the migration:</div><div class="">virsh migrate --verbose --live --abort-on-error --tunnelled --p2p --auto-converge --copy-storage-inc --xml vm-6160.xml 6160 qemu+tls://<destination_hypervisor>/system</div><div class=""><br class=""></div><div class="">Here is the log output, look at the time elapsed:</div><div class=""><br class=""></div><div class=""><div class="">root@virt-hv009:~# virsh domjobinfo 6160</div><div class="">Job type: Unbounded </div><div class="">Time elapsed: 27518 ms</div><div class="">Data processed: 21.506 GiB</div><div class="">Data remaining: 29.003 GiB</div><div class="">Data total: 50.509 GiB</div><div class="">Memory processed: 0.000 B</div><div class="">Memory remaining: 520.820 MiB</div><div class="">Memory total: 520.820 MiB</div><div class="">File processed: 21.506 GiB</div><div class="">File remaining: 28.494 GiB</div><div class="">File total: 50.000 GiB</div><div class="">Constant pages: 0 </div><div class="">Normal pages: 0 </div><div class="">Normal data: 0.000 B</div><div class="">Expected downtime: 300 ms</div><div class="">Setup time: 6 ms</div><div class=""><br class=""></div><div class="">root@virt-hv009:~# virsh domjobinfo 6160</div></div><div class=""><div class="">Job type: Unbounded </div><div class="">Time elapsed: 32331 ms</div><div class="">Data processed: 25.475 GiB</div><div class="">Data remaining: 25.034 GiB</div><div class="">Data total: 50.509 GiB</div><div class="">Memory processed: 0.000 B</div><div class="">Memory remaining: 520.820 MiB</div><div class="">Memory total: 520.820 MiB</div><div class="">File processed: 25.475 GiB</div><div class="">File remaining: 24.525 GiB</div><div class="">File total: 50.000 GiB</div><div class="">Constant pages: 0 </div><div class="">Normal pages: 0 </div><div class="">Normal data: 0.000 B</div><div class="">Expected downtime: 300 ms</div><div class="">Setup time: 6 ms</div><div class=""><br class=""></div><div class="">root@virt-hv009:~# virsh domjobinfo 6160</div><div class="">Job type: Unbounded </div><div class=""><b class="">Time elapsed: 49543 ms</b></div><div class="">Data processed: 50.000 GiB</div><div class="">Data remaining: 520.820 MiB</div><div class="">Data total: 50.509 GiB</div><div class="">Memory processed: 0.000 B</div><div class="">Memory remaining: 520.820 MiB</div><div class="">Memory total: 520.820 MiB</div><div class="">File processed: 50.000 GiB</div><div class="">File remaining: 0.000 B</div><div class="">File total: 50.000 GiB</div><div class="">Constant pages: 0 </div><div class="">Normal pages: 0 </div><div class="">Normal data: 0.000 B</div><div class="">Expected downtime: 300 ms</div><div class="">Setup time: 6 ms</div></div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">^^^^ Here after 49 sec, the disk has been process, but the memory hasn't started</div><div class=""><br class=""></div><div class="">I skip the logs, but below is the output 5minutes & 56 seconds later, still nothing:</div><div class=""><br class=""></div><div class=""><div class="">root@virt-hv009:~# virsh domjobinfo 6160</div><div class="">Job type: Unbounded </div><div class=""><b class="">Time elapsed: 356919 ms</b></div><div class="">Data processed: 50.000 GiB</div><div class="">Data remaining: 520.820 MiB</div><div class="">Data total: 50.509 GiB</div><div class="">Memory processed: 0.000 B</div><div class="">Memory remaining: 520.820 MiB</div><div class="">Memory total: 520.820 MiB</div><div class="">File processed: 50.000 GiB</div><div class="">File remaining: 0.000 B</div><div class="">File total: 50.000 GiB</div><div class="">Constant pages: 0 </div><div class="">Normal pages: 0 </div><div class="">Normal data: 0.000 B</div><div class="">Expected downtime: 300 ms</div><div class="">Setup time: 6 ms</div><div class=""><br class=""></div><div class="">Just after it started to move the memory, but 6 minutes later!</div><div class=""><br class=""></div><div class="">root@virt-hv009:~# virsh domjobinfo 6160</div><div class="">Job type: Unbounded </div><div class=""><b class="">Time elapsed: 360092 ms</b></div><div class="">Data processed: 50.052 GiB</div><div class="">Data remaining: 453.895 MiB</div><div class="">Data total: 50.509 GiB</div><div class="">Memory processed: 53.224 MiB</div><div class="">Memory remaining: 453.895 MiB</div><div class="">Memory total: 520.820 MiB</div><div class="">Memory bandwidth: 98.172 MiB/s</div><div class="">File processed: 50.000 GiB</div><div class="">File remaining: 0.000 B</div><div class="">File total: 50.000 GiB</div><div class="">Constant pages: 3541 </div><div class="">Normal pages: 13591 </div><div class="">Normal data: 53.090 MiB</div><div class="">Expected downtime: 300 ms</div><div class="">Setup time: 6 ms</div><div class=""><br class=""></div><div class="">root@virt-hv009:~# virsh domjobinfo 6160</div><div class="">error: failed to get domain '6160'</div><div class="">error: Domain not found: no domain with matching name '6160'</div></div><div class=""><br class=""></div><div class="">Here the migration is done. But what made it wait more than 5 minutes to start moving the memory? Does anyone knows? Or have any idea what could produce this latency.</div><div class=""><br class=""></div><div class="">- Marco</div></body></html>