[Libguestfs] [PATCH nbdkit v2] PROPOSED: server: Implement list_exports.

[Richard W.M. Jones]

On Wed, Jul 29, 2020 at 09:22:31PM -0500, Eric Blake wrote:
> On 7/22/20 7:42 AM, Richard W.M. Jones wrote:
> >See also:
> >https://www.redhat.com/archives/libguestfs/2020-July/msg00090.html
> >---
> >  docs/nbdkit-plugin.pod   | 57 +++++++++++++++++++++++++++++++++++++---
> >  docs/nbdkit-protocol.pod |  7 +++--
> >  2 files changed, 58 insertions(+), 6 deletions(-)
> 
> I'm playing with code for an initial implementation of this, and
> quickly ran into a design issue.  For plugins, your proposal is
> straightforward.  But for filters, we have several cases:
> 
> 1. Filter doesn't care about export name, so it passes through
> .list_exports to the plugin and advertises the plugin's name; then
> it further passes through the client's export name to .open()
> 
> 2. Filter is providing its own export names, so it skips the
> plugin's .list_exports altogether, and eventually calls .open("") to
> get the plugin's default export.  But where does the filter get the
> information on what exports exist?  For example, the tar filter
> would want to use next_ops->pread to determine what files are
> contained within the tar served by the underlying plugin.  Except
> that .list_exports is called prior to .open, so we have no plugin
> handle or next_ops to use for pread.

I envisaged it would need to get this as a filter parameter from the
command line, IOW from the user.  (Or I suppose if not supplied, then
assume "").

> 3. The filter wants to provide a cartesian product of export names
> (for each export offered by the plugin, the filter then determines a
> sublist of items from that plugin content), which implies multiple
> .plugin opens (in this mode, we probably DO want to pursue my
> proposal in an earlier thread about enhancing the NBD protocol to
> add an NBD_OPT_LIST_HIER and NBD_REP_SERVER_DIR to allow
> hierarchical traversal rather than flat listing of every possible
> export at once).

This one sounds complicated ...

> For 1), the interface for the filter is like any other interface -
> provide a hook for filters to call, along with a next_ops to defer
> into the plugin.  But for 2) I'm thinking we need some way to make
> it easy to write a filter that shares a single underlying connection
> to a plugin among multiple connections to the plugin; and for 3),
> that extends even further into more than one plugin .open for a
> single filter .list_exports.

Could use the same mechanism as filter background threads?
This is currently unsolved but see:
https://github.com/libguestfs/nbdkit/blob/a510aae84a7a8d9cd9ead824fbfac8e0000e85b5/TODO#L71

Those are my thoughts so far, will have to come back to this
email to give a more detailed response later.

Rich.

> For comparison, I'm envisioning something like the current nbd
> plugin having a --shared mode; by default (--shared=false), we get:
> 
> client1 -> nbdkit nbd .open() -> server connection 1
> client2 -> nbdkit nbd .open() -> server connection 2
> 
> but with --shared=true, we get:
> 
> client1 -\
>           > nbdkit nbd .after_fork() -> single server connection
> client2 -/
> 
> We need something similar for a filter to be able to request a
> single shared .open of the underlying plugin (shared by all
> connections into the filter), and which has a lifetime visible
> between the filter's .after_fork and .unload regardless of how many
> .open/.prepare/.close client connections come and go, rather than
> having to repeat a next_ops->open for each filter .open.  Kind of
> the opposite of the reopen filter (which uses multiple calls into
> next_ops->open from a single .open to the filter).
> 
> While filters like tar and ext2 will probably benefit from doing
> explicit sharing of a single underlying plugin, it would also allow
> us to create a new --filter=shared that can be thrown in front of
> any plugin (rendering the existing 'nbdkit nbd --shared' obsolete in
> favor of 'nbdkit nbd --filter=shared').
> 
> Basically, I'm thinking we need the following filter-only functions:
> 
> int nbdkit_plugin_open(int readonly, struct nbdkit_next_ops **next_ops,
>                        void **nxdata);
> 
> which a filter can call at any point between .after_fork and .unload
> to open a distinct connection into the plugin independent of any
> filter clients, and which returns by reference a next_ops/nxdata
> pair that the filter can then use in place of waiting for a
> per-connection next_ops/nxdata passed through
> .prepare/.can_write/.pread/etc.
> 
> void nbdkit_plugin_close (struct nbdkit_next_ops *next_ops,
>                           void *nxdata);
> 
> which a filter later calls to close its own connection into the
> underlying plugin.  Any plugin that uses these new functions would
> probably implement a .open that does NOT call next_open(nxdata),
> because it instead reuses the shared handle that it opened globally.
> 
> -- 
> Eric Blake, Principal Software Engineer
> Red Hat, Inc.           +1-919-301-3226
> Virtualization:  qemu.org | libvirt.org

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW