investigate locks on a domain

[Peter Krempa]

On Tue, Jul 20, 2021 at 15:52:58 +0800, Jiatong Shen wrote:
> Hello community,
> 
> I am seeing following log in production,
> 
> 2021-07-20 07:43:49.417+0000: 3918294: error :
> qemuDomainObjBeginJobInternal:4945 : Timed out during operation: cannot
> acquire state change lock (held by qemuProcessReconnect)
> 2021-07-20 07:44:19.424+0000: 3918296: warning :
> qemuDomainObjBeginJobInternal:4933 : Cannot start job (modify, none) for
> domain instance-0000074e; current job is (modify, none) owned by (3919429
> qemuProcessReconnect, 0 <null>) for (2183193s, 0s)
> 2021-07-20 07:44:19.424+0000: 3918296: error :
> qemuDomainObjBeginJobInternal:4945 : Timed out during operation: cannot
> acquire state change lock (held by qemuProcessReconnect)
> 2021-07-20 07:44:49.428+0000: 3918296: warning :
> qemuDomainObjBeginJobInternal:4933 : Cannot start job (query, none) for
> domain instance-0000074e; current job is (modify, none) owned by (3919429
> qemuProcessReconnect, 0 <null>) for (2183223s, 0s)
> 2021-07-20 07:44:49.428+0000: 3918296: error :
> qemuDomainObjBeginJobInternal:4945 : Timed out during operation: cannot
> acquire state change lock (held by qemuProcessReconnect)
> 2021-07-20 07:45:19.429+0000: 3918298: warning :
> qemuDomainObjBeginJobInternal:4933 :
> 
> I am confused about what is qemuProcessReconnect and why it acquires a
> domain state lock..

qemuProcessReconnect is an operation that is executed in a separate
thread which re-establishes connection to a qemu process once you
restart libvirtd.

This usually means that the reconnection process got stuck for some
reason. Unfortunately your log doesn't show why and unless you've got a
debug log prior to that happening it won't be possible.

Theoretically seeing which function the thread doing the reconnect is
stuck at could perhaps show why.

Usually the only fix for this is to destroy the VM which is stuck.