[libvirt] Failed to terminate process 1275 with SIGTERM: Device or resource busy

[Kashyap Chamarthy]

On Mon, Jan 18, 2016 at 04:19:58PM +0000, Richard W.M. Jones wrote:
> On Mon, Jan 18, 2016 at 03:33:25PM +0000, Richard W.M. Jones wrote:
> > I tried another workaround which was to get virt-resize to fsync the
> > output file before closing the libvirt connection, but that doesn't
> > work for reasons I don't understand so far - still studying this.
> 
> I worked out what was happening here -- I'd inserted the fsync at the
> wrong place in virt-resize.  So I have now successfully worked around
> this for the virt-resize case, however it's still a problem that could
> manifest itself in other uses of libvirt + qemu + slow devices.

We've seen the "Failed to terminate process 1275 with SIGTERM: Device or
resource busy" error occur in context of OpenStack as well[1][2].

The behavior is from virDomainDestroy() API (src/libvirt-domain.c):

    [...]
    * virDomainDestroy first requests that a guest terminate (e.g.
    * SIGTERM), then waits for it to comply. After a reasonable timeout,
    * if the guest still exists, virDomainDestroy will forcefully
    * terminate the guest (e.g. SIGKILL) if necessary (which may produce
    * undesirable results, for example unflushed disk cache in the
    * guest). To avoid this possibility, it's recommended to instead
    * call virDomainDestroyFlags, sending the
    * VIR_DOMAIN_DESTROY_GRACEFUL flag.
    [...]

Dan Berrange explains[1]:

  There are two reasons why you'd get this failure ("Failed to terminate
  process: Device or resource busy") from libvirt. 
   
    - The host is so overloaded that the kernel was not able to clean up
      the process in the time that libvirt was prepared to wait. If this
      is the case, the process should eventually go away on its own
      after a short while longer and everything should return to normal

    - There is some problem, causing the process to get stuck in an
      uninterruptable wait state. This is usually due to something going
      wrong in the storage stack, causing some I/O read/write operation
      to hang in kernel space. In this case the process will stay around
      in the zombie state forever, or until the storage problem is
      resolved.


[1] https://bugzilla.redhat.com/show_bug.cgi?id=1205647 --
    nova.virt.libvirt.driver fails to shutdown reboot instance with
    error 'Code=38 Error=Failed to terminate process 4260 with SIGKILL:
    Device or resource busy' 
[2] https://bugs.launchpad.net/nova/+bug/1353939 -- Rescue fails with
    'Failed to terminate process: Device or resource busy' in the n-cpu
    log

-- 
/kashyap