[Libguestfs] Extending the nbdkit python plugin

[Eric Blake]

On 11/21/19 7:57 AM, Richard W.M. Jones wrote:
> We have an nbdkit plugin that lets you write NBD servers in Python.
> An example of an existing Python plugin is here:
> 
>    https://github.com/libguestfs/nbdkit/blob/master/plugins/python/example.py#L1
> 
> This morning I tried to modify the plugin to use the newer nbdkit API
> (version 2).  One of the things that would change would be passing
> flags parameters to some functions, eg:
> 
>    def pwrite (h, buf, offset):
> 
> might become one of these possibilities (where flags is a bitmask):
> 
>    def pwrite (h, buf, offset, flags):
>    def pwrite (h, buf, offset, flags=0):
> 
> The problem is if we did this it would break all existing Python
> plugins.  While we don't guarantee the nbdkit API for non-C languages,
> we do nevertheless have Python plugins that we care about such as the
> rhv-upload-plugin used by virt-v2v.  Having a flag day which breaks
> all existing plugins is very awkward.

At one point, I tried using Python introspection to determine which 
signature the user's function has, to determine if the user wrote 
against the v1 protocol. It may also be possible to use python's keyword 
argument parsing to pass all optional arguments like flags by key=value 
pairs (allowing future addition of additional flags).  I'll have to 
revive my patches.

> 
> I tried to simply pass the extra arguments from the C code to the
> Python code, and existing plugins break with:
> 
> nbdkit: python[1]: error: ./test.py: pwrite: error: pwrite() takes 3 positional arguments but 4 were given
> 
> One possibility is that we could introspect the plugin to find out how
> many parameters it takes.  This is possible, but very difficult from
> C.  (See https://stackoverflow.com/a/41188411 for how to do it from
> Python).
> 
> Another possibility is we could encourage existing Python plugins to
> add **kwargs to all functions where we might plausibly add extra
> parameters in future, ie. the above would become:
> 
>    def pwrite (h, buf, offset, **kwargs):
> 
> This still requires all Python plugins to change, but at least they
> would remain backwards compatible with old and new nbdkit.

Yes, that's along the lines of the work I had attempted previously.  But 
I attempted it back when we were using Python2, so I'd have to rework it 
for Python3.

>  However I
> couldn't actually work out how to make this work because:
> 
>    >>> def test(a, **kwargs):
>    ...   pass
>    ...
>    >>> test(1)
>    >>> test(1,2)
>    Traceback (most recent call last):
>      File "<stdin>", line 1, in <module>
>    TypeError: test() takes 1 positional argument but 2 were given
> 
> Yet another possibility is the Python plugin itself should declare
> which version of the API it wants, and the C code can then pass the
> correct parameters.  (This is in fact how nbdkit C plugins work).
> This pushes a bunch of work into the C code, but I guess we can deal
> with that.
> 
> So really this is a question directed at Python experts.  What's the
> right way to go about this?

I'll try and revive my patches; I may be able to save you some time 
based on what I already have in my tree.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org