<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=gb2312"> <meta name="Generator" content="Microsoft Word 12 (filtered medium)"> <style></style> </head> <body lang="ZH-CN" link="blue" vlink="purple" style="text-justify-trim:punctuation"> <div class="WordSection1"> Hi everyone,<o:p></o:p> <o:p> </o:p> I do ram migration operation in KVM environment(libvirt1.2.4 qemu1.5.1).<o:p></o:p> I encountered libvirtd deadlock or segmentfault when I destroy the<o:p></o:p> migration VM on destination.<o:p></o:p> I got the problem by flowing steps:<o:p></o:p> step 1: migrate VM.<o:p></o:p> step 2: execute "virsh destroy [VMName]" to destroy the migration VM on <o:p></o:p> destination immediately.<o:p></o:p> step 3: the destination libvirtd will be probably deadlock or segmentfault.<o:p></o:p> <o:p> </o:p> Deadlock stack as followed:<o:p></o:p> #0 0x00007fb5c18132d4 in __lll_lock_wait () from /lib64/libpthread.so.0<o:p></o:p> #1 0x00007fb5c180e659 in _L_lock_1008 () from /lib64/libpthread.so.0<o:p></o:p> #2 0x00007fb5c180e46e in pthread_mutex_lock () from /lib64/libpthread.so.0<o:p></o:p> #3 0x00007fb5c45d175f in virMutexLock (m=0x7fb5b0066ed0) at util/virthread.c:88<o:p></o:p> #4 0x00007fb5c45b6b04 in virObjectLock (anyobj=0x7fb5b0066ec0) at<o:p></o:p> util/virobject.c:323<o:p></o:p> #5 0x00007fb5b8f4842a in qemuMonitorEmitEvent (mon=0x7fb5b0066ec0,<o:p></o:p> event=0x7fb5b00688d0 "SHUTDOWN", seconds=1399374472, micros=509994,<o:p></o:p> details=0x0) at qemu/qemu_monitor.c:1185<o:p></o:p> #6 0x00007fb5b8f62af2 in qemuMonitorJSONIOProcessEvent (mon=0x7fb5b0066ec0,<o:p></o:p> obj=0x7fb5b0069080) at qemu/qemu_monitor_json.c:158<o:p></o:p> #7 0x00007fb5b8f62d25 in qemuMonitorJSONIOProcessLine (mon=0x7fb5b0066ec0,<o:p></o:p> line=0x7fb5b005bbe0 "{\"timestamp\": {\"seconds\": 1399374472,<o:p></o:p> \"microseconds\": 509994}, \"event\": \"SHUTDOWN\"}",msg=0x7fb5bd873c80)<o:p></o:p> at qemu/qemu_monitor_json.c:195<o:p></o:p> #8 0x00007fb5b8f62f85 in qemuMonitorJSONIOProcess (mon=0x7fb5b0066ec0,<o:p></o:p> data=0x7fb5b0060770 "{\"timestamp\": {\"seconds\": 1399374472,<o:p></o:p> \"microseconds\": 509994},\"event\": \"SHUTDOWN\"}\r\n", len=85,<o:p></o:p> msg=0x7fb5bd873c80) at qemu/qemu_monitor_json.c:237<o:p></o:p> #9 0x00007fb5b8f49aa0 in qemuMonitorIOProcess (mon=0x7fb5b0066ec0)<o:p></o:p> at qemu/qemu_monitor.c:402<o:p></o:p> #10 0x00007fb5b8f4a09b in qemuMonitorIO (watch=20, fd=24, events=0,<o:p></o:p> opaque=0x7fb5b0066ec0) at qemu/qemu_monitor.c:651<o:p></o:p> #11 0x00007fb5c458c4d9 in virEventPollDispatchHandles (nfds=17, fds=0x7fb5b0068a60)<o:p></o:p> at util/vireventpoll.c:510<o:p></o:p> #12 0x00007fb5c458decf in virEventPollRunOnce () at util/vireventpoll.c:659<o:p></o:p> #13 0x00007fb5c458bfcc in virEventRunDefaultImpl () at util/virevent.c:308<o:p></o:p> #14 0x00007fb5c51a17a9 in virNetServerRun (srv=0x7fb5c5411d70)<o:p></o:p> at rpc/virnetserver.c:1139<o:p></o:p> #15 0x00007fb5c5157f63 in main (argc=3, argv=0x7fff7fc04f48) at libvirtd.c:150<o:p></o:p> <o:p> </o:p> <o:p> </o:p> After analysis, I found it may be caused by multithreaded simultaneously <o:p></o:p> access to the global variables "vm->privateData->mon".<o:p></o:p> When problems occur，there are three libvirtd threads at work on destination<o:p></o:p> host，suppose：<o:p></o:p> ThreadA: migration thread，do qemuProcessStart.<o:p></o:p> ThreadB: destroy thread，do qemuDoaminDestroy -> qemuProcessStop.<o:p></o:p> ThreadC：Monitor Thread，do IOWrite、IORead and some other operations according to<o:p></o:p> the mon->msg when mon->fd change. When threadB destroy happpens, this thread would<o:p></o:p> handle the SUHTDOWN event.<o:p></o:p> <o:p> </o:p> In threadA, when it sends QMP command to Qemu, it will operate the vm->privateData->mon<o:p></o:p> lock. Such as the operation "qemuDomainObjEnterMonitor -> qemuMonitorSetBalloon -><o:p></o:p> qemuDomainObjExitMonitor", but it's not an atomic operation. If "virsh destroy [VMName]"<o:p></o:p> happens during this operation, threadB will set the lock vm->privateData->mon to NULL in<o:p></o:p> qemuProcessStop. And then in threadA, the function qemuDomainObjExitMonitor will fail to<o:p></o:p> unlock vm->privateData->mon as it's NULL. So, threadC will never acquire the<o:p></o:p> vm->privateData->mon lock and the deadlock problem happened.<o:p></o:p> <o:p> </o:p> what was worse, if qemuMonitorSetBalloon perform succeed in threadA.<o:p></o:p> ThreadA will coutinue to execute till it enter the function qemuMonitorSetDomainLog,<o:p></o:p> it would cause segmentfault at VIR_FORCE_CLOSE(mon->logfd) due to mon is NULL.<o:p></o:p> <o:p> </o:p> I could not find a good way to sovle this problem. Does anyone have good ideas? <o:p></o:p> Thanks.<o:p></o:p> <o:p> </o:p> Ps:<o:p></o:p> I find an easy way to reproduce this problem more probably by following steps:<o:p></o:p> <o:p> </o:p> step 1: Fault Injection, fit into this patch and update the libvirtd on destination host:<o:p></o:p> --- src/qemu/qemu_process.c 2014-05-06 19:06:00.000000000 +0800<o:p></o:p> +++ src/qemu/qemu_process.c 2014-05-06 19:07:12.000000000 +0800<o:p></o:p> @@ -4131,6 +4131,8 @@<o:p></o:p> vm->def->mem.cur_balloon);<o:p></o:p> goto cleanup;<o:p></o:p> }<o:p></o:p> + VIR_DEBUG("Fault Injection, sleep 3 seconds.");<o:p></o:p> + sleep(3);<o:p></o:p> qemuDomainObjEnterMonitor(driver, vm);<o:p></o:p> if (vm->def->memballoon && vm->def->memballoon->period)<o:p></o:p> qemuMonitorSetMemoryStatsPeriod(priv->mon, vm->def->memballoon->period);<o:p></o:p> <o:p> </o:p> step 2: migrate VM.<o:p></o:p> step 3: execute "virsh destroy [VMName]" to destroy the migration VM on destination<o:p></o:p> when log prints "Fault Injection, sleep 3 seconds."<o:p></o:p> step 4: the libvirtd deadlock stack occurs as above mentioned.<o:p></o:p> <o:p> </o:p> Regards<o:p></o:p> </div> </body> </html>