[libvirt-users] VM crash and lock manager

[Franky Van Liedekerke]

On Tue, 29 Mar 2016 19:27:40 +0200
villeneu at kassis.univ-brest.fr wrote:

> Hello
> 
> I changed my hypervisors ( 8)  from centos 6.0 to fedora 23 and I
> added locks to prevent starting VM on multiple nodes, and it's work
> well.
> 
> Since I changed the OS some old VMs   ( centos 5.x and 6.x ) with very
> old kernel seem very instable and crash very often ( after one or two
> days ). Before they never have these problems.
> 
>  I saw, perhaps the reason is the dev is too slow ???  see screenshot 
> joined
> 
> A major problem with these crashs is that  the VMs couldn't be
> destroyed by the virsh command, the qemu process is notified as
> defunct by the ps command.
> 
> with virsh destroy VM
> I often get
> 
> Failed to terminate process xxx with SIGTERM: Device or resource busy
> and the VM is still in the list
> 
> If I try to remove all the file associated with the VM in /var/run or
> /var/lib....channel .. It doesn't give results The
> VM is still running as defunct and after a libvirt restart , libvirt
> and virsh command are pending.
> 
> With sanlock it's a problem because the is never release and it's
> impossible to restart the VM. I can't remove the VM in the sanlock
> list. I tried kill commands, rm ... but without a real success.
> 
> I tried many command with sanlock perhaps I misunderstood some
> commands but I never succeed in removing lock on the VMs.
> 
> sanlock client rem_lockspace -r
> __LIBVIRT__DISKS__:b90b9c61e2d6413077205907ffb3281a:/var/lib/libvirt/images/POOL_ADMIN/sanlock/b90b9c61e2d6413077205907ffb3281a:0:4
> -p 9383
> 
> So I'm trying to use lock_manager to replace the sanlock but
> lock_manager doesn't have tools to retrieve the lock / VM name in
> case of failure or crash.
> 
> My questions are, is there a way to release or to force to release a
> lock with sanlock even if the qemu process is defunc and a way to
> restart libvirt in normal usage ?
> 
> Perhaps by the API it`s possible to forece the release the lock ?
> 
> Is there tools to retreive or calculate SHA256 lock for a VM with the
> lock_manager lock. I saw it's the lock is a SHA256 hash of the path
> and the VM name but I nerver get the good hash when I try by the hand.
> 
> Is there tools to retreive or calculate SHA256 the lock for a VM with
> the lock_manager lock. I saw that the  lock is a SHA256 hash of the
> path and the VM name but I never get the good hash when I try by the
> hand.
> 
> Thanks.
> Sorry for my poor english ... I 'am french
> Michel

It seems the lock held by sanlock isn't being released, so the vm can't
be shut down properly. Maybe some selinux issue?
I've been using virtlockd without troubles for more than a year already
(without selinux). For the locks, see if the lslocks command helps you.

Franky