<html> <head> <meta http-equiv="Content-Type" content="text/html; charset=us-ascii"> <meta name="Generator" content="Microsoft Exchange Server">  <style></style> </head> <body> <font face="Calibri" size="2"><span style="font-size:10.5pt;"> <div align="left" style="text-align:justify;">Hello,</div> <div align="left" style="text-indent:21.6pt;text-align:justify;">I found a problem that libvirtd on destination crash frequently while migrating vms concurrently. For example, if I migrate 10 vms concurrently ceaselessly, then after about 30 minutes the libvirtd on destination will crash. So I analyzed and found two bugs during migration process.</div> <div align="left" style="text-indent:21.6pt;text-align:justify;">First, during migration prepare phase on destination, libvirtd assigns ports to qemu to be startd on destination. But the port increase operation is not aomic, so there’s a chance that multi vms get the same port, and only the first one can start successfully, others will fail to start. I’ve applied a patch to solve this bug, and I test it, it works well. If only this bug exists, libvirtd will not crash. The second bug is fatal.</div> <div align="left" style="text-indent:21.6pt;text-align:justify;">Second, I found the libvirtd crash because of segment fault which is produced by accessing vm released. Apparently it’s caused by multi-thread operation, thread A access vm data which has released by thread B. At last I proved my thought right.</div> <div align="left" style="text-align:justify;"><font face="Times New Roman"> </font></div> <div align="left" style="text-indent:21.6pt;text-align:justify;">Step 1. Because of bug one, the port is already occupied, so qemu on destination failed to start and sent a HANGUP signal to libvirtd, then libvirtd received this VIR_EVENT_HANDLE_HANGUP event, thread A dealing with events called qemuProcessHandleMonitorEOF as following:</div> <div align="left" style="text-indent:21.6pt;text-align:justify;"><font face="Times New Roman"> </font></div> <table width="703" style="width:421.8pt;margin-left:21pt;"> <col width="703" style="width:421.8pt;"> <tr> <td align="left" style="text-align:justify;"><font face="Times New Roman" size="2"><span style="font-size:10pt;">#0 <font color="red">qemuProcessHandleMonitorEOF</font> (mon=0x7f4dcd9c3130, vm=0x7f4dcd9c9780) <div> at qemu/qemu_process.c:399</div> <div>#1 0x00007f4dc18d9e87 in qemuMonitorIO (watch=68, fd=27, events=8,</div> <div> opaque=0x7f4dcd9c3130) at qemu/qemu_monitor.c:668</div> <div>#2 0x00007f4dccae6604 in virEventPollDispatchHandles (nfds=18,</div> <div> fds=0x7f4db4017e70) at util/vireventpoll.c:500</div> <div>#3 0x00007f4dccae7ff2 in virEventPollRunOnce () at util/vireventpoll.c:646</div> <div>#4 0x00007f4dccae60e4 in virEventRunDefaultImpl () at util/virevent.c:273</div> <div>#5 0x00007f4dccc40b25 in virNetServerRun (srv=0x7f4dcd8d26b0)</div> <div> at rpc/virnetserver.c:1106</div> <div>#6 0x00007f4dcd6164c9 in main (argc=3, argv=0x7fff8d8f9f88)</div> <div> at libvirtd.c:1518</div> </span></font></td> </tr> </table> <div align="left" style="text-indent:21.6pt;text-align:justify;"><font face="Times New Roman"> </font></div> <div align="left" style="text-indent:21.6pt;text-align:justify;"><font face="Times New Roman"> </font></div> <table width="703" style="width:421.8pt;margin-left:21pt;"> <col width="703" style="width:421.8pt;"> <tr> <td align="left" style="text-align:justify;"><font face="Times New Roman" size="2"><span style="font-size:10pt;">static int <b>virEventPollDispatchHandles</b>(int nfds, struct pollfd *fds) { <div> <font face="宋体">……</font></div> <div> <font color="red">/*</font><font color="red"> </font></div> <div style="text-indent:40pt;"><font color="red">deleted flag is still false now, so we pass through to qemuProcessHandleMonitorEOF </font></div> <div style="text-indent:40pt;"><font color="red">*/</font></div> <div> if (eventLoop.handles[i].deleted) {</div> <div> EVENT_DEBUG("Skip deleted n=%d w=%d f=%d", i,</div> <div> eventLoop.handles[i].watch, eventLoop.handles[i].fd);</div> <div> continue;</div> <div> }</div> </span></font></td> </tr> </table> <div align="left" style="text-indent:21.6pt;text-align:justify;"><font face="Times New Roman"> </font></div> <div align="left" style="text-indent:21.6pt;text-align:justify;"><font face="Times New Roman"> </font></div> <div align="left" style="text-indent:21.6pt;text-align:justify;">Step 2: Thread B dealing with migration on destination set deleted flag in virEventPollRemoveHandle as following:</div> <div align="left" style="text-indent:21.6pt;text-align:justify;"><font face="Times New Roman"> </font></div> <table width="703" style="width:421.8pt;margin-left:21pt;"> <col width="703" style="width:421.8pt;"> <tr> <td align="left" style="text-align:justify;"><font face="Times New Roman" size="2"><span style="font-size:10pt;">#0 <font color="red">virEventPollRemoveHandle</font> (watch=74) at util/vireventpoll.c:176 <div>#1 0x00007f4dccae5e6f in <font color="red">virEventRemoveHandle</font> (watch=74)</div> <div> at util/virevent.c:97</div> <div>#2 0x00007f4dc18d8ca8 in <font color="red">qemuMonitorClose</font> (mon=0x7f4dbc030910)</div> <div> at qemu/qemu_monitor.c:831</div> <div>#3 0x00007f4dc18bec63 in <font color="red">qemuProcessStop</font> (driver=0x7f4dcd9bd400,</div> <div> vm=0x7f4dbc00ed20, reason=VIR_DOMAIN_SHUTOFF_FAILED, flags=0)</div> <div> at qemu/qemu_process.c:4302</div> <div>#4 0x00007f4dc18c1a83 in <font color="red">qemuProcessStart</font> (conn=0x7f4dbc031020,</div> <div> driver=0x7f4dcd9bd400, vm=0x7f4dbc00ed20,</div> <div> migrateFrom=0x7f4dbc01af90 "tcp:[::]:49152", stdin_fd=-1,</div> <div> stdin_path=0x0, snapshot=0x0,</div> <div> vmop=VIR_NETDEV_VPORT_PROFILE_OP_MIGRATE_IN_START, flags=6)</div> <div> at qemu/qemu_process.c:4145</div> <div>#5 0x00007f4dc18cc688 in qemuMigrationPrepareAny (driver=0x7f4dcd9bd400,</div> </span></font></td> </tr> </table> <div align="left" style="text-align:justify;"><font face="Times New Roman"> </font></div> <div align="left" style="text-indent:21.6pt;text-align:justify;">Step 3: Thread B cleanup vm in qemuMigrationPrepareAny after qemuProcessStart failed.</div> <div align="left" style="text-indent:21.6pt;text-align:justify;"><font face="Times New Roman"> </font></div> <table width="703" style="width:421.8pt;margin-left:21pt;"> <col width="703" style="width:421.8pt;"> <tr> <td align="left" style="text-align:justify;"><font face="Times New Roman" size="2"><span style="font-size:10pt;">#0 <font color="red">virDomainObjDispose</font> (obj=0x7f4dcd9c9780) at conf/domain_conf.c:2009 <div>#1 0x00007f4dccb0ccd9 in <font color="red">virObjectUnref</font> (anyobj=0x7f4dcd9c9780)</div> <div> at util/virobject.c:266</div> <div>#2 0x00007f4dccb42340 in <font color="red">virDomainObjListRemove</font> (doms=0x7f4dcd9bd4f0,</div> <div> dom=0x7f4dcd9c9780) at conf/domain_conf.c:2342</div> <div>#3 0x00007f4dc189ac33 in <font color="red">qemuDomainRemoveInactive</font> (driver=0x7f4dcd9bd400,</div> <div> vm=0x7f4dcd9c9780) at qemu/qemu_domain.c:1993</div> <div>#4 0x00007f4dc18ccad5 in <font color="red">qemuMigrationPrepareAny</font> (driver=0x7f4dcd9bd400,</div> </span></font></td> </tr> </table> <div align="left" style="text-indent:21.6pt;text-align:justify;"><font face="Times New Roman"> </font></div> <div align="left" style="text-indent:21.6pt;text-align:justify;">Step 4: Thread A access priv which is released by thread B before, then libvirtd crash, bomb!</div> <div align="left" style="text-indent:21.6pt;text-align:justify;"><font face="Times New Roman"> </font></div> <table width="703" style="width:421.8pt;margin-left:21pt;"> <col width="703" style="width:421.8pt;"> <tr> <td align="left" style="text-align:justify;"><font face="Times New Roman" size="2"><span style="font-size:10pt;">static void <div><b>qemuProcessHandleMonitorEOF</b>(qemuMonitorPtr mon ATTRIBUTE_UNUSED,</div> <div> virDomainObjPtr vm)</div> <div>{</div> <div> virQEMUDriverPtr driver = qemu_driver;</div> <div> virDomainEventPtr event = NULL;</div> <div> qemuDomainObjPrivatePtr priv;</div> <div> int eventReason = VIR_DOMAIN_EVENT_STOPPED_SHUTDOWN;</div> <div> int stopReason = VIR_DOMAIN_SHUTOFF_SHUTDOWN;</div> <div> const char *auditReason = "shutdown";</div> <div> </div> <div> VIR_DEBUG("Received EOF on %p '%s'", vm, vm->def->name);</div> <div> </div> <div> virObjectLock(vm);</div> <div> </div> <div> <font color="red">priv </font>= vm->privateData;</div> <div><span style="background-color:yellow;">(gdb) p priv</span></div> <div><span style="background-color:yellow;">$1 = (qemuDomainObjPrivatePtr) </span><font color="red"><span style="background-color:yellow;">0x0</span></font></div> <div> if (priv->beingDestroyed) {</div> </span></font></td> </tr> </table> <div align="left" style="text-align:justify;"><font face="Times New Roman"> </font></div> <div align="left" style="text-align:justify;"> At last if anything bad happened to make <font color="red">qemuProcessStart</font><font color="red"> </font>failed during migration on destination, we’ll be in the big trouble that accessing some memory freed. I didn’t find any locks or flags exist could stop this happening. Please help me out, thanks a lot.</div> <div align="left" style="text-align:justify;"><font face="Times New Roman"> </font></div> <div>Best Regards,</div> <div>-WangYufei</div> <div align="left" style="text-align:justify;"><font face="Times New Roman"> </font></div> <div align="left" style="text-align:justify;"><font face="Times New Roman"> </font></div> <div align="left" style="text-align:justify;"><font face="Times New Roman"> </font></div> </span></font> </body> </html>