Network disks and replacing qemu-block-curl|ssh with nbdkit

[Richard W.M. Jones]

On Thu, Apr 14, 2022 at 05:02:46PM -0500, Jonathon Jongsma wrote:
> 1. secrets
[...]
> Fortunately, nbdkit provides a method for reading cookies and
> passwords from a file, which should be secure if the file has
> permissions set properly. So I'm currently planning to write a file
> containing the cookies and pass them to nbdkit by specifying the
> filename. But I'm still confused about the username/password
> possibility.

You can also send the password or cookie over an inherited file
descriptor, which has the possible advantage that the secret will
never hit the disk at all.

For completeness I should say that we found HTTP authentication
against some servers to be quite slow (presumably because validating a
password involves a lot of machinery so doing it on every request is
slow).  For those servers we implemented a complicated scheme where
you could make an authenticated request, fetch the cookie that the
server sends back, send back the cookie, _and_ autorenew the cookie if
it times out.  (Did I say this was complicated?)  This is required for
at least VMware servers and Docker registries.

https://libguestfs.org/nbdkit-curl-plugin.1.html#HEADER-AND-COOKIE-SCRIPTS

I wouldn't try implementing this through libvirt ...

> 2. readahead
>
> 3. blockdev-create

See also:
https://listman.redhat.com/archives/libguestfs/2022-April/028674.html

I agree we should implement creation for ssh disks (not sure if it's
possible or even makes sense for curl).  Shouldn't be too difficult.

You might also want to think about VDDK disk support, ie. is it
possible to make the nbdkit stuff generic enough that
nbdkit-vddk-plugin can be slotted in later?  It would allow a libvirt
domain to be backed with remote disks stored on VMFS, or to use
VMware's own proprietary drivers to open local VMDK files, both
significant enhancements.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
nbdkit - Flexible, fast NBD server with plugins
https://gitlab.com/nbdkit/nbdkit