<html> <head> <meta http-equiv="content-type" content="text/html; charset=UTF-8"> </head> <body> <tt>Dear libvirt community,</tt><tt> </tt><tt> </tt><tt> </tt><tt><span style="font-size: 12px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: left; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;">Using recent Ubuntu Stein Cloud Packages, we are observing random libvirtd live-migration crashes on the target host.</tt><tt> </tt><tt><span style="font-size: 12px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: left; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"> Libvirt is having a SEGFAULT with the qemu driver. Transferring block devices usually works without issues. </tt><tt> </tt><tt><span style="font-size: 12px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: left; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;">However, the following memory transfer is causing the target libvirtd randomly to close down its socket, resulting in a roll-backed migration process.</tt><tt> I can reproduce this with large VMs, which have a large memory pool. The last error message we see in libvirt logs is: error : qemuMonitorJSONCommandWithFd:315 : internal error: Missing monitor reply object With this, libvirt segfaults and restarts. Before we encountered this issue, we used an older nova-compute package (19.0.3). Not sure if this made a difference with usage of libvirtd-api. After upgrade, we also see a lot of recurring errors during migration: warning : qemuDomainObjBeginJobInternal:7044 : Cannot start job (query, none, none) for domain instance-00008f56; current job is (none, none, migration in) owned by (0 <null>, 0 <null>, 0 remoteDispatchDomainMigratePrepare3Params (flags=0x809b)) for (0s, 0s, 14834s) error : qemuDomainObjBeginJobInternal:7066 : Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchDomainMigratePrepare3Params) They don't abort the running migration process, but spam every minute to the systemd journal. Source and destination run the same packages: </tt><tt>Ubuntu 18.04.4 LTS (GNU/Linux 4.15.0-99-generic x86_64)</tt><tt> </tt><tt>OpenStack Stein (Ubuntu Cloud Archive)</tt><tt> </tt><tt>Libvirt+QEMU_x86</tt><tt> </tt><tt>keystone-common 2:15.0.1-0ubuntu1~cloud0</tt><tt> </tt><tt>libvirt-daemon 5.0.0-1ubuntu2.6~cloud0</tt><tt> </tt><tt>qemu-system-x86 1:3.1+dfsg-2ubuntu3.7~cloud0</tt><tt> </tt><tt>neutron-linuxbridge-agent 2:14.2.0-0ubuntu1~cloud0</tt><tt> </tt><tt>neutron-plugin-ml2 2:14.2.0-0ubuntu1~cloud0</tt><tt> </tt><tt>nova-compute 2:19.2.0-0ubuntu1~cloud0</tt><tt> </tt><tt>nova-compute-libvirt 2:19.2.0-0ubuntu1~cloud0</tt><tt> </tt><tt> </tt><tt><span style="font-size: 12px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: left; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"></tt> <tt><span style="font-size: 12px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: left; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;">I have attached source/destination debug logs from libvirtd and nova-compute here:</tt> <tt><span style="font-size: 12px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: left; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"><a class="moz-txt-link-freetext" href="https://denzelx.ddns.net/index.php/s/KPJ7vv4aTcb69XD">https://denzelx.ddns.net/index.php/s/KPJ7vv4aTcb69XD</a> </tt> <tt><span style="font-size: 12px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: left; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;">Any help would be nice!</tt> <tt><span style="font-size: 12px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: left; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"> </tt> <tt><span style="font-size: 12px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: left; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;">Best Regards</tt> <pre class="moz-signature" cols="72">-- M.Sc Alex Walender de.NBI Cloud Bielefeld Administrator Center for Biotechnology (CeBiTec) University of Bielefeld 33594 Bielefeld Germany room: M3-118 phone: +49 (521) 106 2907</pre> </body> </html>