[Libguestfs] nbdkit background threads

[Eric Blake]

On 2/10/20 1:39 PM, Richard W.M. Jones wrote:
> https://github.com/libguestfs/nbdkit/blob/ecef5b16359fb5af7e7abf4fd2fb3ad5438b16be/TODO#L76
> 
> Already existing filters (readahead, cache) could be improved if
> filters could open a background work thread or threads which connect
> independently to the plugin.  A proposed new filter (scan) cannot
> really be written at all without this capability.
> 
> First of all the reason this can't be done today is because filters
> are called with a next_ops structure which is only valid transiently
> during the filter callback.  It cannot safely be saved or passed to
> another thread.
> (https://github.com/libguestfs/nbdkit/blob/master/docs/nbdkit-filter.pod#next-plugin)

I was independently thinking of fixing this: we want next_ops to be have 
a life equal to the connection itself, rather than stack-allocated.  I'm 
hoping to post a patch for that shortly, as part of my experimentation 
with implementing ext2 as a filter instead of a plugin, where ext2 has 
the limitation that when writing a custom io_manager, you only get ONE 
spot where you can pass in an opaque pointer: our .prepare will have to 
pass a long-lived nxdata to ext2fs_open().

But that's only a partial solution towards the rest of your goal of 
having threading available.

> 
> The seemingly obvious implementation - which is what I tried today -
> would be to let filters create background threads in .config_complete.
> We would provide a filter API something like:
> 
>   int nbdkit_open_connection (struct nbdkit_next_ops **next_ops,
>                               void **nxdata,
> 			     /* needs a close function */);
> 
> It would return a next_ops and nxdata that the filter could then use
> to make data calls from the background thread into the underlying
> layers.
> 
> To ensure safe unloading of filters and plugins we would also need a
> new filter callback (which I called .bg_kill) which must close all
> background connections opened by the filter before returning.

Seems reasonable.

> 
> I believe from my rough implementation that this is feasible.  However
> I also thought about another way we might do this: We might open a
> loopback socket (eg. socketpair) which is passed to the filter and
> which the filter connects to using libnbd.  nbdkit internally would
> treat this as if it was a regular external connection.  Of course this
> would require libnbd as a dependency, or disable filters / features if
> not available.

There's also the question of how we would do compostion.  For example, 
right now, the 'split' plugin basically acts as a compose operation over 
the 'file' plugin (you can concatenate more than one regular file), but 
more generically, we'd want some way to compose an arbitrary number of 
plugins and/or filters, where having a socketpair to each such composed 
child lets us handle more than just files.

The 'nbd' plugin already has a dependency on libnbd, but allowing 
socketpair background threading would make this dependency extend to 
much more than that one plugin.

> 
> Any thoughts on this?
> 
> Rich.
> 

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org