[libvirt] Failed to terminate process 1275 with SIGTERM: Device or resource busy

[Daniel P. Berrange]

On Tue, Jan 19, 2016 at 01:39:41PM +0000, Richard W.M. Jones wrote:
> On Tue, Jan 19, 2016 at 12:31:48PM +0100, Kashyap Chamarthy wrote:
> > On Mon, Jan 18, 2016 at 04:19:58PM +0000, Richard W.M. Jones wrote:
> > > On Mon, Jan 18, 2016 at 03:33:25PM +0000, Richard W.M. Jones wrote:
> > > > I tried another workaround which was to get virt-resize to fsync the
> > > > output file before closing the libvirt connection, but that doesn't
> > > > work for reasons I don't understand so far - still studying this.
> > > 
> > > I worked out what was happening here -- I'd inserted the fsync at the
> > > wrong place in virt-resize.  So I have now successfully worked around
> > > this for the virt-resize case, however it's still a problem that could
> > > manifest itself in other uses of libvirt + qemu + slow devices.
> > 
> > We've seen the "Failed to terminate process 1275 with SIGTERM: Device or
> > resource busy" error occur in context of OpenStack as well[1][2].
> > 
> > The behavior is from virDomainDestroy() API (src/libvirt-domain.c):
> > 
> >     [...]
> >     * virDomainDestroy first requests that a guest terminate (e.g.
> >     * SIGTERM), then waits for it to comply. After a reasonable timeout,
> >     * if the guest still exists, virDomainDestroy will forcefully
> >     * terminate the guest (e.g. SIGKILL) if necessary (which may produce
> >     * undesirable results, for example unflushed disk cache in the
> >     * guest). To avoid this possibility, it's recommended to instead
> >     * call virDomainDestroyFlags, sending the
> >     * VIR_DOMAIN_DESTROY_GRACEFUL flag.
> >     [...]
> > 
> > Dan Berrange explains[1]:
> > 
> >   There are two reasons why you'd get this failure ("Failed to terminate
> >   process: Device or resource busy") from libvirt. 
> >    
> >     - The host is so overloaded that the kernel was not able to clean up
> >       the process in the time that libvirt was prepared to wait. If this
> >       is the case, the process should eventually go away on its own
> >       after a short while longer and everything should return to normal
> > 
> >     - There is some problem, causing the process to get stuck in an
> >       uninterruptable wait state. This is usually due to something going
> >       wrong in the storage stack, causing some I/O read/write operation
> >       to hang in kernel space. In this case the process will stay around
> >       in the zombie state forever, or until the storage problem is
> >       resolved.
> 
> Thanks for finding this documentation.
> 
> The problem with this theory is we are passing the
> VIR_DOMAIN_DESTROY_GRACEFUL flag, so that would indicate that this
> flag is buggy.

I don't think it does. Passing GRACEFUL flag means libvirt will try
/less/ hard to kill QEMU, so it is /more/ likely that you will get
the

  "Failed to terminate process 1275 with SIGTERM: Device or resource busy"

In general, that error message is something to be expected from the
virDomainDestroy() API, as we won't wait for death forever. If an
app wishes to wait forever, they should either re-issue the destroy
API call, or wait for an event notification of VIR_DOMAIN_EVENT_STOPPED
to arrive.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|