[libvirt-users] Could not destroy domain, current job is remoteDispatchConnectGetAllDomainStats

[Michal Privoznik]

On 01/17/2018 03:45 PM, Serhii Kharchenko wrote:
> Hello libvirt-users list,
> 
> We're catching the same bug since 3.4.0 version (3.3.0 works OK).
> So, we have process that is permanently connected to libvirtd via socket
> and it is collecting stats, listening to events and control the VPSes.
> 
> When we try to 'shutdown' a number of VPSes we often catch the bug. One of
> VPSes sticks in 'in shutdown' state, no related 'qemu' process is present,
> and there is the next error in the log:
> 
> Jan 17 13:54:20 server1 libvirtd[20437]: 2018-01-17 13:54:20.005+0000:
> 20438: warning : qemuGetProcessInfo:1460 : cannot parse process status data
> Jan 17 13:54:20 server1 libvirtd[20437]: 2018-01-17 13:54:20.006+0000:
> 20441: error : virFileReadAll:1420 : Failed to open file
> '/sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d36\x2dDOMAIN1.scope/cpuacct.usage':
> No such file or directory
> Jan 17 13:54:20 server1 libvirtd[20437]: 2018-01-17 13:54:20.006+0000:
> 20441: error : virCgroupGetValueStr:844 : Unable to read from
> '/sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d36\x2dDOMAIN1.scope/cpuacct.usage':
> No such file or directory
> Jan 17 13:54:20 server1 libvirtd[20437]: 2018-01-17 13:54:20.006+0000:
> 20441: error : virCgroupGetDomainTotalCpuStats:3319 : unable to get cpu
> account: Operation not permitted
> Jan 17 13:54:23 server1 libvirtd[20437]: 2018-01-17 13:54:23.805+0000:
> 20522: warning : qemuDomainObjBeginJobInternal:4862 : Cannot start job
> (destroy, none) for domain DOMAIN1; current job is (query, none) owned by
> (20440 remoteDispatchConnectGetAllDomainStats, 0 <null>) for (30s, 0s)
> Jan 17 13:54:23 server1 libvirtd[20437]: 2018-01-17 13:54:23.805+0000:
> 20522: error : qemuDomainObjBeginJobInternal:4874 : Timed out during
> operation: cannot acquire state change lock (held by
> remoteDispatchConnectGetAllDomainStats)

Sounds like qemuConnectGetAllDomainStats() forgot to unset the job on a
domain. Can you please attach gdb to libvirtd and run 't a a bt' and
share the output?

Michal