[Libguestfs] [PATCH nbdkit filters-v2 2/5] Introduce filters.

[Richard W.M. Jones]

On Fri, Jan 19, 2018 at 01:38:36PM -0600, Eric Blake wrote:
> > Right, but this patch is against the version 1 API.  We will need to
> > add flags at some point.
> 
> So, do we ever want filters to be usable via version 1 instead of
> version 2 backend API, or should we be aiming that all filters are
> automatically version 2 from the point where we release this?

Yes, the latter.  We can add flags, "breaking" ABI only briefly (if we
want a filters ABI guarantee at all that is).

> >> Would it ever make sense to allow a filter to intentionally request a
> >> lower concurrency level than the underlying plugin?  At connection time,
> >> you'd compute the threading level as the lowest level requested by any
> >> element of the chain; it might make it easier to write plugins that
> >> aren't fully parallel (such filters may penalize the speed that can
> >> otherwise be achieved by using the plugin in isolation - but again, it
> >> may be useful for testing).  That would imply adding a
> >> backend.thread_model() callback, which needs to be exposed to filters
> >> but not to plugins (plugins continue to use the #define at compile
> >> time); if the filter does not define the .thread_model callback, it
> >> defaults to fully-parallel; but if it does define the callback, it can
> >> result in something less concurrent.
> > 
> > Yes, this is a good idea actually.
> 
> Are you planning on writing that, or do you want me to take a shot at it?

There's some more churn in this patch series so it's better if I do
this.  I'm going to try to post an updated series soon.  The partition
filter almost works ...

> > I couldn't think of a case where we'd need the same filter multiple
> > times.  It could be made to work but we'd have to ban global
> > variables and modify the load() function.
> 
> I can (sort of) see it if we support composition; but again, if
> composition is done right, you still have just a single use of each
> plugin per nbdkit process.  Let's say I want to access [0-100M) and
> [200M-300M) of a given file via composition.  Here's a possible way to
> write it:
> 
> nbdkit --filter=compose compose='--filter=offset offset=0 range=100M
> file file=FILE' --filter=offset offset=200M range=100M file file=FILE
> 
> Is there a way to rewrite it so I don't have to nest quotes when doing
> more than one composition?  If there is, then --filter=offset appears
> twice, as does 'file file=FILE'.  But either way, under the hood, we'd
> implement it as two separate nbdkit processes if composition is a filter:
> 
> nbdkit --> compose --> offset 0/100 --> file
>                    \-> nbd ==> nbdkit --> offset 200/100 --> file
[complicated stuff]

Sounds complicated.

In that diagram you've got two file plugins.  I guess you were
planning to fork to cope with that, but I think it's better if we
support composition by exposing the NBD client to plugins and
connecting multiple actual nbdkit processes together.

I think a real use case might be supporting RAID across files ...

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/