[libvirt] the error is bug？

[Michal Privoznik]

On 29.05.2015 16:38, cloudbuglist wrote:
> Hi dear!
>   Today I came across a problem, the following is a log and software versions.
> soft version:
> os:centos 6.5
> libvirt:0.10.29
> qemu:0.12.415
> 
> 
> 2015-05-28 03:28:58.001+0000: 20919: error : qemuDomainObjBeginJobInternal:843 : Timed out during operation: cannot acquire state change lock
> 2015-05-28 03:30:37.000+0000: 20918: warning : qemuDomainObjBeginJobInternal:839 : Cannot start job (query, none) for domain i-2-437-VM; current job is (query, none) owned by (20917, 0)
> 2015-05-28 03:30:37.000+0000: 20918: error : qemuDomainObjBeginJobInternal:843 : Timed out during operation: cannot acquire state change lock
> 2015-05-28 03:31:01.000+0000: 20919: warning : qemuDomainObjBeginJobInternal:839 : Cannot start job (query, none) for domain i-2-437-VM; current job is (query, none) owned by (20917, 0)
> 2015-05-28 03:31:01.000+0000: 20919: error : qemuDomainObjBeginJobInternal:843 : Timed out during operation: cannot acquire state change lock
> 2015-05-28 03:32:40.000+0000: 20921: warning : qemuDomainObjBeginJobInternal:839 : Cannot start job (query, none) for domain i-2-437-VM; current job is (query, none) owned by (20917, 0)
> 2015-05-28 03:32:40.000+0000: 20921: error : qemuDomainObjBeginJobInternal:843 : Timed out during operation: cannot acquire state change lock
> 2015-05-28 03:33:06.000+0000: 20918: warning : qemuDomainObjBeginJobInternal:839 : Cannot start job (query, none) for domain i-2-437-VM; current job is (query, none) owned by (20917, 0)
> 2015-05-28 03:33:06.000+0000: 20918: error : qemuDomainObjBeginJobInternal:843 : Timed out during operation: cannot acquire state change lock
> 2015-05-28 03:33:42.009+0000: 20916: error : virNetSocketReadWire:1194 : End of file while reading data: Input/output error
> 2015-05-28 03:34:02.000+0000: 20921: warning : qemuDomainObjBeginJobInternal:839 : Cannot start job (query, none) for domain i-2-437-VM; current job is (query, none) owned by (20917, 0)
> 2015-05-28 03:34:02.000+0000: 20921: error : qemuDomainObjBeginJobInternal:843 : Timed out during operation: cannot acquire state change lock
> 2015-05-28 03:34:19.001+0000: 20920: warning : qemuDomainObjBeginJobInternal:839 : Cannot start job (query, none) for domain i-2-437-VM; current job is (query, none) owned by (20917, 0)
> 2015-05-28 03:34:19.001+0000: 20920: error : qemuDomainObjBeginJobInternal:843 : Timed out during operation: cannot acquire state change lock
> 2015-05-28 03:34:43.384+0000: 20916: error : virNetSocketReadWire:1194 : End of file while reading data: Input/output error
> 2015-05-28 03:34:47.000+0000: 20918: warning : qemuDomainObjBeginJobInternal:839 : Cannot start job (query, none) for domain i-2-437-VM; current job is (query, none) owned by (20917, 0)
> 2015-05-28 03:34:47.000+0000: 20918: error : qemuDomainObjBeginJobInternal:843 : Timed out during operation: cannot acquire state change lock
> 2015-05-28 03:34:52.000+0000: 20920: warning : qemuDomainObjBeginJobInternal:839 : Cannot start job (query, none) for domain i-2-437-VM; current job is (query, none) owned by (20917, 0)
> 2015-05-28 03:34:52.000+0000: 20920: error : qemuDomainObjBeginJobInternal:843 : Timed out during operation: cannot acquire state change lock
> 2015-05-28 03:35:09.001+0000: 20921: warning : qemuDomainObjBeginJobInternal:839 : Cannot start job (query, none) for domain i-2-437-VM; current job is (query, none) owned by (20917, 0)
> 2015-05-28 03:35:09.001+0000: 20921: error : qemuDomainObjBeginJobInternal:843 : Timed out during operation: cannot acquire state change lock

This usually means, that the other thread (20917) has locked the domain
for changes (or to be more technical, it set a job). If you attach gdb
onto the daemon and query what is the other thread doing, you'll get the
picture. Unfortunately, some operations on a domain can take ages to
finish. The other possibility could be, that a thread has set the job
and accessed an NFS backed storage which went stale. Or this can as well
be a bug. Hard to tell from only this info. Also - does this happen with
new libvirt? 0.10.29 is a bit old today.

Michal