[Libguestfs] nbdkit background threads

[Eric Blake]

On 2/11/20 2:41 AM, Richard W.M. Jones wrote:
> On Mon, Feb 10, 2020 at 01:52:25PM -0600, Eric Blake wrote:
>> On 2/10/20 1:39 PM, Richard W.M. Jones wrote:
>>> https://github.com/libguestfs/nbdkit/blob/ecef5b16359fb5af7e7abf4fd2fb3ad5438b16be/TODO#L76
>>>
>>> Already existing filters (readahead, cache) could be improved if
>>> filters could open a background work thread or threads which connect
>>> independently to the plugin.  A proposed new filter (scan) cannot
>>> really be written at all without this capability.
>>>
>>> First of all the reason this can't be done today is because filters
>>> are called with a next_ops structure which is only valid transiently
>>> during the filter callback.  It cannot safely be saved or passed to
>>> another thread.
>>> (https://github.com/libguestfs/nbdkit/blob/master/docs/nbdkit-filter.pod#next-plugin)
>>
>> I was independently thinking of fixing this: we want next_ops to be
>> have a life equal to the connection itself, rather than
>> stack-allocated.  I'm hoping to post a patch for that shortly, as
>> part of my experimentation with implementing ext2 as a filter
>> instead of a plugin, where ext2 has the limitation that when writing
>> a custom io_manager, you only get ONE spot where you can pass in an
>> opaque pointer: our .prepare will have to pass a long-lived nxdata
>> to ext2fs_open().
> 
> Yes I can see this could be tricky, especially in the presence of
> multiple threads serving a single connection.  The only alternative
> way I can think to do it would be to use a pointer in thread-local
> storage which would would update with next_ops / nxdata each time.
> 
> Do you have a preliminary patch for this yet, or do you want me to
> look at it today?

Just sent the stable nxdata patch.  I'm still working on the ext2 filter 
patches, though.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org