<div dir="ltr">Ok, I'll check it out. <div> </div><div>Look like your patch is still not in master branch。May you commit it? I'll check out it and test for force console case</div> <div class="gmail_quote"><div dir="ltr" class="gmail_attr">Michal Privoznik <<a href="mailto:mprivozn@redhat.com" target="_blank">mprivozn@redhat.com</a>> 于2019年11月25日周一下午10:12写道： </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On 11/25/19 11:53 AM, Lance Liu wrote: > We first produce this bug in rhel7.4's libvir daemon。For easily > produce the bug, the step can be as follows: 1. add sleep(3) in > daemonStreamFilter() pre virMutexLock(&stream->priv->lock), then > build libvirtd bin executable, then restart libvirtd 2. use virsh > console open one vm's console, for this console（the vm's kernel need > console=ttyS0 boot parameter,then just input from keyboard on and on > 3. use virsh console --force to break the previous console session, > than you will get libvirt daemon deadlock。 > > And for the problem client->privData to be released problem, only > virNetServerClientClose() will free client->privData and client, I > think this can be fixed by remove virNetServerClientClose() from > daemonStreamEvent(), and replace with > virNetServerClientImmediateClose(), so virNetServerProcessClients() > will test the session would be closed。 RHEL-7.4 you say? That's libvirt-3.2.0 which is far from what we have on the master. Also, RHEL-7.4 itself is kind of old as RHEL-7.7 was released during summer. At least that explains why your original patch was against older libvirt. When I revert your patch I can see the deadlock still happening: Thread 5 (Thread 0x7feac8936700 (LWP 162328)): #0 0x00007feacd1d6f6c in __lll_lock_wait () at /lib64/libpthread.so.0 #1 0x00007feacd1cf194 in pthread_mutex_lock () at /lib64/libpthread.so.0 #2 0x00007feacdc78a9e in virMutexLock (m=0x565030303700) at ../../src/util/virthread.c:79 #3 0x00007feacdc488a2 in virObjectLock (anyobj=0x5650303036f0) at ../../src/util/virobject.c:433 #4 0x00007feacdd9f8bc in virNetServerClientSendMessage (client=0x5650303036f0, msg=0x7feab40281e0) at ../../src/rpc/virnetserverclient.c:1532 #5 0x00007feacdd9ac27 in virNetServerProgramSendError (program=536903814, version=1, client=0x5650303036f0, msg=0x7feab40281e0, rerr=0x7feac8935730, procedure=201, type=3, serial=16) at ../../src/rpc/virnetserverprogram.c:168 #6 0x00007feacdd9ad52 in virNetServerProgramSendStreamError (prog=0x5650302ccb70, client=0x5650303036f0, msg=0x7feab40281e0, rerr=0x7feac8935730, procedure=201, serial=16) at ../../src/rpc/virnetserverprogram.c:217 #7 0x000056502e9bbd24 in daemonStreamEvent (st=0x7feac40022a0, events=4, opaque=0x5650303036f0) at ../../src/remote/remote_daemon_stream.c:256 #8 0x00007feacdc01648 in virFDStreamCloseInt (st=0x7feac40022a0, streamAbort=true) at ../../src/util/virfdstream.c:715 #9 0x00007feacdc017d1 in virFDStreamAbort (st=0x7feac40022a0) at ../../src/util/virfdstream.c:759 #10 0x00007feacded9e1d in virStreamAbort (stream=0x7feac40022a0) at ../../src/libvirt-stream.c:1237 #11 0x00007feacdd456e3 in virChrdevOpen (devs=0x7fea740cae00, source=0x7fea740d26a0, st=0x7feab402f220, force=true) at ../../src/conf/virchrdev.c:387 #12 0x00007feab3f722e7 in qemuDomainOpenConsole (dom=0x7feab40329b0, dev_name=0x0, st=0x7feab402f220, flags=1) at ../../src/qemu/qemu_driver.c:17309 #13 0x00007feacdeb375f in virDomainOpenConsole (dom=0x7feab40329b0, dev_name=0x0, st=0x7feab402f220, flags=1) at ../../src/libvirt-domain.c:9662 #14 0x000056502e999894 in remoteDispatchDomainOpenConsole (server=0x56503027fc30, client=0x56503031ad70, msg=0x565030304eb0, rerr=0x7feac8935ad0, args=0x7feab402cd70) at ./remote/remote_daemon_dispatch_stubs.h:9211 #15 0x000056502e99976f in remoteDispatchDomainOpenConsoleHelper (server=0x56503027fc30, client=0x56503031ad70, msg=0x565030304eb0, rerr=0x7feac8935ad0, args=0x7feab402cd70, ret=0x0) at ./remote/remote_daemon_dispatch_stubs.h:9178 #16 0x00007feacdd9b4b1 in virNetServerProgramDispatchCall (prog=0x5650302ccb70, server=0x56503027fc30, client=0x56503031ad70, msg=0x565030304eb0) at ../../src/rpc/virnetserverprogram.c:430 #17 0x00007feacdd9b026 in virNetServerProgramDispatch (prog=0x5650302ccb70, server=0x56503027fc30, client=0x56503031ad70, msg=0x565030304eb0) at ../../src/rpc/virnetserverprogram.c:302 #18 0x00007feacdda1e34 in virNetServerProcessMsg (srv=0x56503027fc30, client=0x56503031ad70, prog=0x5650302ccb70, msg=0x565030304eb0) at ../../src/rpc/virnetserver.c:136 #19 0x00007feacdda1ef4 in virNetServerHandleJob (jobOpaque=0x565030336ea0, opaque=0x56503027fc30) at ../../src/rpc/virnetserver.c:153 #20 0x00007feacdc79800 in virThreadPoolWorker (opaque=0x56503028a1f0) at ../../src/util/virthreadpool.c:163 #21 0x00007feacdc78dad in virThreadHelper (data=0x56503028afd0) at ../../src/util/virthread.c:196 #22 0x00007feacd1cc458 in start_thread () at /lib64/libpthread.so.0 #23 0x00007feacd0fa6ef in clone () at /lib64/libc.so.6 Thread 1 (Thread 0x7feaca8a7dc0 (LWP 162311)): #0 0x00007feacd1d6f6c in __lll_lock_wait () at /lib64/libpthread.so.0 #1 0x00007feacd1cf194 in pthread_mutex_lock () at /lib64/libpthread.so.0 #2 0x00007feacdc78a9e in virMutexLock (m=0x5650302ff110) at ../../src/util/virthread.c:79 #3 0x000056502e9bbde3 in daemonStreamFilter (client=0x5650303036f0, msg=0x5650303097d0, opaque=0x7feac4003020) at ../../src/remote/remote_daemon_stream.c:297 #4 0x00007feacdd9f03b in virNetServerClientDispatchRead (client=0x5650303036f0) at ../../src/rpc/virnetserverclient.c:1288 #5 0x00007feacdd9f6ab in virNetServerClientDispatchEvent (sock=0x565030309a90, events=1, opaque=0x5650303036f0) at ../../src/rpc/virnetserverclient.c:1485 #6 0x00007feacdd907f9 in virNetSocketEventHandle (watch=19, fd=28, events=1, opaque=0x565030309a90) at ../../src/rpc/virnetsocket.c:2177 #7 0x00007feacdbff32d in virEventPollDispatchHandles (nfds=16, fds=0x5650302bf200) at ../../src/util/vireventpoll.c:503 #8 0x00007feacdbffb86 in virEventPollRunOnce () at ../../src/util/vireventpoll.c:658 #9 0x00007feacdbfdafb in virEventRunDefaultImpl () at ../../src/util/virevent.c:322 #10 0x00007feacdda1a0d in virNetDaemonRun (dmn=0x56503027f880) at ../../src/rpc/virnetdaemon.c:836 #11 0x000056502e98843f in main (argc=2, argv=0x7ffc308703a8) at ../../src/remote/remote_daemon.c:1427 But I think I have an idea how to fix this. Will post patches later today. Michal </blockquote></div></div>