[Libguestfs] [PATCH nbdkit filters-v2 2/5] Introduce filters.

[Eric Blake]

On 01/19/2018 11:12 AM, Richard W.M. Jones wrote:

>>> +
>>> +=head2 C<.pread>
>>> +
>>> + int (*pread) (struct nbdkit_next_ops *next_ops, void *nxdata,
>>> +               void *handle, void *buf, uint32_t count, uint64_t offset);
>>
>> Wrong signature, missing the uint32_t flags that the backend interface
>> has, and that I'm adding for plugins API_VERSION 2 for FUA support.
> 
> Right, but this patch is against the version 1 API.  We will need to
> add flags at some point.

So, do we ever want filters to be usable via version 1 instead of
version 2 backend API, or should we be aiming that all filters are
automatically version 2 from the point where we release this?  I'm fine
if we have some churn in .git by taking your patches with version 1
signatures then my FUA patches on top that converts to version 2; I'm
more worried that if we allow filters with version 1 semantics that we
may run into some interesting difficulties (not the least of which is
that a version 2 plugin that can support efficient FUA is inherently
penalized if it is used with a filter talking only version 1).


>>> +Filters may have to use pthread primitives like mutexes to achieve
>>> +this.
>>
>> Would it ever make sense to allow a filter to intentionally request a
>> lower concurrency level than the underlying plugin?  At connection time,
>> you'd compute the threading level as the lowest level requested by any
>> element of the chain; it might make it easier to write plugins that
>> aren't fully parallel (such filters may penalize the speed that can
>> otherwise be achieved by using the plugin in isolation - but again, it
>> may be useful for testing).  That would imply adding a
>> backend.thread_model() callback, which needs to be exposed to filters
>> but not to plugins (plugins continue to use the #define at compile
>> time); if the filter does not define the .thread_model callback, it
>> defaults to fully-parallel; but if it does define the callback, it can
>> result in something less concurrent.
> 
> Yes, this is a good idea actually.

Are you planning on writing that, or do you want me to take a shot at it?

> 
>>> +=item B<--filter> FILTER
>>> +
>>> +Add a filter before the plugin.  This option may be given one or more
>>> +times to stack filters in front of the plugin.  They are processed in
>>> +the order they appear on the command line.  See L</FILTERS> and
>>> +L<nbdkit-filter(3)>.
>>> +
>>
>> Does it ever make sense to provide the same filter more than once on the
>> command line?  Or are we stating that a given filter can only be used
>> once in the chain?
> 
> I couldn't think of a case where we'd need the same filter multiple
> times.  It could be made to work but we'd have to ban global
> variables and modify the load() function.

I can (sort of) see it if we support composition; but again, if
composition is done right, you still have just a single use of each
plugin per nbdkit process.  Let's say I want to access [0-100M) and
[200M-300M) of a given file via composition.  Here's a possible way to
write it:

nbdkit --filter=compose compose='--filter=offset offset=0 range=100M
file file=FILE' --filter=offset offset=200M range=100M file file=FILE

Is there a way to rewrite it so I don't have to nest quotes when doing
more than one composition?  If there is, then --filter=offset appears
twice, as does 'file file=FILE'.  But either way, under the hood, we'd
implement it as two separate nbdkit processes if composition is a filter:

nbdkit --> compose --> offset 0/100 --> file
                   \-> nbd ==> nbdkit --> offset 200/100 --> file

where -> is the backend chain, and ==> is an NBD socket.

Or maybe composition is a plugin, in which case it might look more like:

nbdkit compose sub='--filter=offset offset=0 range=100M file file=FILE'
sub='--filter=offset offset=200M range=100M file file=FILE'

(Can config even accept the same key= with different values more than
once on the command line?)

nbdkit --> compose ==> nbdkit --> offset 0/100 --> file
                   \=> nbdkit --> offset 200/100 --> file

Certainly some tradeoffs to revisit depending on whether composition
fits better as a filter or a plugin

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 619 bytes
Desc: OpenPGP digital signature
URL: <http://listman.redhat.com/archives/libguestfs/attachments/20180119/4432d55d/attachment.sig>