[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [Libguestfs] Thoughts on nbdkit automatic reconnection



On 9/18/19 7:59 AM, Richard W.M. Jones wrote:
> We have a running problem with the nbdkit VDDK plugin where the VDDK
> side apparently disconnects or the network connection is interrupted.
> During a virt-v2v conversion this causes the entire operation to fail,
> and since v2v conversions take many hours that's not a happy outcome.
> 
> (Aside: I should say that we see many cases where it's claimed that
> the connection was dropped, but often when we examine them in detail
> the cause is something else.  But it seems like this disconnection
> thing does happen sometimes.)

nbdkit is not alone - qemu is currently trying to add patches to add nbd
reconnect:

https://lists.gnu.org/archive/html/qemu-devel/2019-09/msg03621.html

> 
> To put this isn't concrete terms which don't involve v2v, let's say
> you were doing something like:
> 
>   nbdkit ssh host=remote /var/tmp/test.iso \
>     --run 'qemu-img convert -p -f raw $nbd -O qcow2 test.qcow2'
> 
> which copies a file over ssh to local.  If /var/tmp/test.iso is very
> large and/or the connection is very slow, and the network connection
> is interrupted then the whole operation fails.  If nbdkit could
> retry/reconnect on failure then the operation might succeed.
> 
> There are lots of parameters associated with retrying, eg:
> 
>  - how often should you retry before giving up?
> 
>  - how long should you wait between retries?
> 
>  - which errors should cause a retry, which are a hard failure?

- do you want TCP keepalive active during the session

> 
> So I had an idea we could implement this as a generic "retry" filter,
> like:
> 
>   nbdkit ssh ... --filter=retry retries=5 retry-delay=5 retry-exponential=yes

Interesting idea.

> 
> This cannot be implemented with the current design of filters because
> a filter would have to call the plugin .close and .open methods, but
> filters don't have access to those from regular data functions, and in
> any case this would cause a new plugin handle to be allocated.

Our .open handling is already odd: we document but do not enforce that a
filter must call next_open on success, but does not necessarily do so on
failure.  Depending on where things fail, it may be possible that we
have a memory leak and/or end up calling .close without a matching
.open; I'm trying to come up with a definitive test case demonstrating
if that is a problem.  I noticed that while trying to make nbdkit return
NBD_REP_ERR_XXX when .open fails, rather than dropping the connection
altogether (since that's a case where a single TCP connection would need
to result in multiple .open/.close pairings).

> We could probably do it if we added a special .reopen method to
> plugins.  We could either require plugins which support the concept of
> retrying to implement this, or we could have a generic implementation
> in server/backend.c which would call .close, .open and cope with the
> new handle.

It sounds like something that only needs to be exposed for filters to
use; I'm having a hard time seeing how a plugin would do it, so keeping
the magic in server/backend.c sounds reasonable.

> Another way to do this would be to modify each plugin to add the
> feature.  nbdkit-nbd-plugin has this for a very limited case, but no
> others do, and it's quite complex to implement in plugins.  As far as
> I can see it involves checking the return value of any data call that
> the plugin makes and performing the reconnection logic, while not
> changing the handle (so just calling self->close, self->open isn't
> going to work).
> 
> If anyone has any thoughts about this I'd be happy to hear them.
> 
> Rich.
> 

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

Attachment: signature.asc
Description: OpenPGP digital signature


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]