<html><head><meta http-equiv="content-type" content="text/html; charset=GB2312"><style>body { line-height: 1.5; }body { font-size: 14px; font-family: 'Microsoft YaHei UI'; color: rgb(0, 0, 0); line-height: 1.5; }</style></head><body> <div><span></span><br></div><div>Hello, </div><div><br></div><div>I have meet one issues while using snapshot-create-as with external file.</div><div><br></div><div>Centos 8, </div><div>Libivirt 7.0.0</div><div>Qemu: 5.2.0</div><div><br></div><div>While making one snapshot with memory and disk extaernal file, and killing the process of the vm qemu same time mostly, <span style="line-height: 1.5; background-color: transparent;">the snapshot-create-as command blocked.</span></div><div><span style="line-height: 1.5; background-color: transparent;">So it will failed to start the vm again for "</span>cannot acquire state change lock (held by monitor=remoteDispatchDomainSnapshotCreateXML)<span style="line-height: 1.5; background-color: transparent;">".</span></div><div><span style="line-height: 1.5; background-color: transparent;"><br></span></div><div><span style="line-height: 1.5; background-color: transparent;">The log is as bellow: </span></div><div><div>2021-08-12 10:27:38.373+0000: 4344: error : qemuDomainAgentAvailable:8125 : Guest agent is not responding: QEMU guest agent is not connected</div><div>2021-08-12 10:27:38.573+0000: 4340: error : qemuDomainAgentAvailable:8125 : Guest agent is not responding: QEMU guest agent is not connected</div><div>2021-08-12 10:27:38.757+0000: 128211: error : qemuDomainAgentAvailable:8125 : Guest agent is not responding: QEMU guest agent is not connected</div><div>2021-08-12 10:27:39.397+0000: 4341: error : qemuDomainAgentAvailable:8125 : Guest agent is not responding: QEMU guest agent is not connected</div><div>2021-08-12 10:27:39.887+0000: 4309: error : virNetSocketReadWire:1817 : End of file while reading data: Input/output error</div><div>2021-08-12 10:27:53.735+0000: 4340: error : qemuDomainAgentAvailable:8125 : Guest agent is not responding: QEMU guest agent is not connected</div><div>2021-08-12 10:27:54.822+0000: 4344: error : qemuDomainAgentAvailable:8125 : Guest agent is not responding: QEMU guest agent is not connected</div><div>2021-08-12 10:27:55.230+0000: 4340: error : qemuDomainAgentAvailable:8125 : Guest agent is not responding: QEMU guest agent is not connected</div><div>2021-08-12 10:46:40.994+0000: 4344: warning : qemuDomainObjTaint:6187 : Domain id=95 name='vm_6_135' uuid=b2812e01-331a-4e65-912b-73be5ace2271 is tainted: high-privileges</div><div>2021-08-12 10:48:05.785+0000: 4309: error : virNetSocketReadWire:1817 : End of file while reading data: Input/output error</div><div>2021-08-12 10:49:50.440+0000: 1643666: error : qemuMonitorIO:583 : internal error: End of file from qemu monitor</div><div>2021-08-12 10:50:13.809+0000: 4341: warning : qemuDomainObjTaint:6187 : Domain id=96 name='vm_6_135' uuid=b2812e01-331a-4e65-912b-73be5ace2271 is tainted: high-privileges</div><div>2021-08-12 10:51:21.587+0000: 1646757: error : qemuMonitorIO:583 : internal error: End of file from qemu monitor</div><div>2021-08-12 10:52:28.421+0000: 4342: warning : qemuDomainObjBeginJobInternal:953 : Cannot start job (none, none, start) for domain vm_6_135; current job is (none, none, snapshot) owned by (0 <null>, 0 <null>, 4340 remoteDispatchDomainSnapshotCreateXML (flags=0x280)) for (0s, 0s, 67s)</div><div>2021-08-12 10:52:28.421+0000: 4342: error : qemuDomainObjBeginJobInternal:975 : Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchDomainSnapshotCreateXML)</div><div>2021-08-12 10:53:10.522+0000: 4342: warning : qemuDomainObjBeginJobInternal:953 : Cannot start job (none, none, start) for domain vm_6_135; current job is (none, none, snapshot) owned by (0 <null>, 0 <null>, 4340 remoteDispatchDomainSnapshotCreateXML (flags=0x280)) for (0s, 0s, 109s)</div><div>2021-08-12 10:53:10.522+0000: 4342: error : qemuDomainObjBeginJobInternal:975 : Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchDomainSnapshotCreateXML)</div><div>2021-08-12 10:53:17.821+0000: 128211: warning : qemuDomainObjBeginJobInternal:953 : Cannot start job (none, none, start) for domain vm_6_135; current job is (none, none, snapshot) owned by (0 <null>, 0 <null>, 4340 remoteDispatchDomainSnapshotCreateXML (flags=0x280)) for (0s, 0s, 117s)</div><div>2021-08-12 10:53:17.821+0000: 128211: error : qemuDomainObjBeginJobInternal:975 : Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchDomainSnapshotCreateXML)</div><div>... ... </div><div>2021-08-13 05:49:41.240+0000: 4342: warning : qemuDomainObjBeginJobInternal:953 : Cannot start job (none, none, start) for domain vm_6_135; current job is (none, none, snapshot) owned by (0 <null>, 0 <null>, 4340 remoteDispatchDomainSnapshotCreateXML (flags=0x280)) for (0s, 0s, 68300s)</div></div><div><br></div><div><br></div><div>Gdb attach the proces of the libvirt, found there are two <span style="font-variant-ligatures: normal; orphans: 2; widows: 2;">suspicious threads</span>, stack as below:</div><div>gdb) bt</div><div>#0 0x00007f8f91a492fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0</div><div>#1 0x00007f8f9574ed7a in virCondWait () from /lib64/libvirt.so.0</div><div>#2 0x00007f8f95784ef5 in virDomainObjWait () from /lib64/libvirt.so.0</div><div>#3 0x00007f8f422eb26d in qemuMigrationSrcWaitForCompletion () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so</div><div>#4 0x00007f8f422ee5d3 in qemuMigrationSrcToFile () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so</div><div>#5 0x00007f8f42331caa in qemuSaveImageCreate () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so</div><div>#6 0x00007f8f42336ac0 in qemuSnapshotCreateXML () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so</div><div>#7 0x00007f8f422b6ac1 in qemuDomainSnapshotCreateXML () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so</div><div>#8 0x00007f8f9591f604 in virDomainSnapshotCreateXML () from /lib64/libvirt.so.0</div><div>#9 0x0000562390841b81 in remoteDispatchDomainSnapshotCreateXMLHelper ()</div><div>#10 0x00007f8f9580e807 in virNetServerProgramDispatch () from /lib64/libvirt.so.0</div><div>#11 0x00007f8f95813b26 in virNetServerHandleJob () from /lib64/libvirt.so.0</div><div>#12 0x00007f8f9574f85f in virThreadPoolWorker () from /lib64/libvirt.so.0</div><div>#13 0x00007f8f9574eecb in virThreadHelper () from /lib64/libvirt.so.0</div><div>#14 0x00007f8f91a4314a in start_thread () from /lib64/libpthread.so.0</div><div><span style="background-color: transparent;">#15 0x00007f8f941f1f23 in clone () from /lib64/libc.so.6</span></div><div> </div><div><div>(gdb) bt</div><div>#0 0x00007f8f91a492fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0</div><div>#1 0x00007f8f9574ed7a in virCondWait () from /lib64/libvirt.so.0</div><div>#2 0x00007f8f95784ef5 in virDomainObjWait () from /lib64/libvirt.so.0</div><div>#3 0x00007f8f42257ac8 in qemuBlockStorageSourceCreateGeneric () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so</div><div>#4 0x00007f8f4225b88d in qemuBlockStorageSourceCreate () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so</div><div>#5 0x00007f8f423346ee in qemuSnapshotDiskPrepareOne () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so</div><div>#6 0x00007f8f423353f8 in qemuSnapshotCreateXML () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so</div><div>#7 0x00007f8f422b6ac1 in qemuDomainSnapshotCreateXML () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so</div><div>#8 0x00007f8f9591f604 in virDomainSnapshotCreateXML () from /lib64/libvirt.so.0</div><div>#9 0x0000562390841b81 in remoteDispatchDomainSnapshotCreateXMLHelper ()</div><div>#10 0x00007f8f9580e807 in virNetServerProgramDispatch () from /lib64/libvirt.so.0</div><div>#11 0x00007f8f95813b26 in virNetServerHandleJob () from /lib64/libvirt.so.0</div><div>#12 0x00007f8f9574f85f in virThreadPoolWorker () from /lib64/libvirt.so.0</div><div>#13 0x00007f8f9574eecb in virThreadHelper () from /lib64/libvirt.so.0</div><div>#14 0x00007f8f91a4314a in start_thread () from /lib64/libpthread.so.0</div><div>#15 0x00007f8f941f1f23 in clone () from /lib64/libc.so.6</div><div>(gdb) </div></div><div><br></div><div><br></div><div>The call work flow one is: </div><div>qemuSnapshotCreateActiveExternal -> qemuSaveImageCreate -> qemuMigrationSrcToFile -> qemuMigrationSrcWaitForCompletion -> virDomainObjWait </div><div><br></div><div>The others is similar :</div><div><span style="color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);"> </span><span style="line-height: 1.5; background-color: transparent;">qemuSnapshotCreateXML</span><span style="line-height: 1.5; background-color: transparent;"> 　－＞　．．．</span></div><div><span style="line-height: 1.5; background-color: transparent;"><br></span></div><div><span style="line-height: 1.5; background-color: transparent;"><br></span></div><div><span style="line-height: 1.5; background-color: transparent;">It may be resume after the restart libvirtd.service.</span></div><div><span style="line-height: 1.5; background-color: transparent;"><br></span></div><div>Is it correct that the stat change lock should be unlocked while the vm is terminal or shutdown and how to fix it?</div><div><br></div><div>Thanks!</div><div><br></div><div><br></div><hr style="width: 210px; height: 1px;" color="#b5c4df" size="1" align="left"> <div><span><div style="MARGIN: 10px; FONT-FAMILY: verdana; FONT-SIZE: 10pt"><div>13520781732@189.cn</div></div></span></div> </body></html>