[libvirt PATCH 0/3] RFC: using nbdkit for network drives in libvirt

[Michal Prívozník]

On 6/22/22 23:26, Jonathon Jongsma wrote:
> Hi guys,
> 
> I've been working on adding support for nbdkit to libvirt for network
> storage sources like http and ssh. See
> https://bugzilla.redhat.com/show_bug.cgi?id=2016527 for more
> information, but the summary is that RHEL does not want to ship the qemu
> storage plugins for curl and ssh.  Handling them outside of the qemu
> process provides several advantages such as reduced attack surface and
> stability.
> 
> I have something that works for me, but as I have not dealt with the
> storage stuff much before, I have a feeling that I'm missing some
> things.
> 
> A quick summary of the code:
>  - at startup I query to see whether nbdkit exists on the host and if
>    so, I query which plugins/filters are installed. This is stored as
>    qemuNbdkitCaps on the qemu driver
>  - When the driver prepares the domain, we go through each disk source
>    and determine whether the nbdkit capabilities allow us to support
>    this disk via nbdkit, and if so, we allocate a qemuNbdkitProcess
>    object and stash it in the private data of the virStorageSource.
>  - The presence or absence of this qemuNbdkitProcess data then indicates
>    whether this disk will be served to qemu indirectly via nbdkit or not
>  - When we launch the qemuProcess, as part of the "external device
>    start" step, I launch a ndkit process for each disk that is supported
>    by nbdkit. I also optionally fork a child process to communicate
>    authentication details and cookies to the nbdkit process via a unix
>    socket.
>  - for devices which are served by the ndkit process, I change the qemu
>    commandline in the following ways:
>    - I no longer pass auth/cookie secrets to qemu (those are handled by
>      nbdkit)
>    - I replace the actual network URL of the remote disk source with the
>      path to the nbdkit unix socket
> 
> Known shortcomings
>  - I don't yet re-query for nbdkit / nbdkit caps, so need to restart libvirt to
>    pick up newly-installed nbdkit or additional capabilities

There are couple of ways to solve this:

1) mimic what we do for dnsmasq: basically, when initializing the bridge
driver we query dnsmasq for caps networkStateInitialize() ->
dnsmasqCapsNewFromBinary() and when needing the caps for real (i.e.
spawning dnsmasq when starting a network) we forcefully refresh the
caps: networkStartDhcpDaemon() -> networkDnsmasqCapsRefresh().

2) use full blown virFileCache() machinery which reacts to mtime of
given binary.

An argument against going with 1) is that while spawning an extra
process (dnsmasq) when starting a network is acceptable (a network is
not started that often), spawning an extra process for each disk (or
even an image in backingchain) might be too expensive.

>  - testing is pretty limited at the moment
>  - selinux not working yet
>  - creating disks isn't supported, though Rich has added some support
>    for that upstream in the nbdkit ssh plugin.
> 
> I'd appreciate feedback on what i've got so far.
> 
> Jonathon Jongsma (3):
>   docs: clarify 'readahead' and 'timeout' for disks
>   schema: Be more flexible for diskSourceNetworkProtocolPropsCommon
>   WIP: use nbdkit for remote disk sources
> 
>  docs/formatdomain.rst                         |  10 +-
>  include/libvirt/virterror.h                   |   1 +
>  po/POTFILES                                   |   1 +
>  src/conf/schemas/domaincommon.rng             |  34 +-
>  src/qemu/meson.build                          |   1 +
>  src/qemu/qemu_block.c                         |  64 +-
>  src/qemu/qemu_block.h                         |   1 +
>  src/qemu/qemu_command.c                       |  26 +-
>  src/qemu/qemu_conf.c                          |  19 +
>  src/qemu/qemu_conf.h                          |   5 +
>  src/qemu/qemu_domain.c                        | 110 ++-
>  src/qemu/qemu_domain.h                        |   5 +
>  src/qemu/qemu_driver.c                        |   4 +-
>  src/qemu/qemu_extdevice.c                     |  25 +
>  src/qemu/qemu_nbdkit.c                        | 629 ++++++++++++++++++
>  src/qemu/qemu_nbdkit.h                        |  89 +++
>  src/qemu/qemu_validate.c                      |  22 +-
>  src/qemu/qemu_validate.h                      |   4 +-
>  src/util/virerror.c                           |   1 +
>  tests/qemublocktest.c                         |   8 +-
>  tests/qemustatusxml2xmldata/modern-in.xml     |   1 -
>  ...sk-cdrom-network-nbdkit.x86_64-latest.args |  42 ++
>  .../disk-cdrom-network-nbdkit.xml             |   1 +
>  ...isk-network-http-nbdkit.x86_64-latest.args |  45 ++
>  .../disk-network-http-nbdkit.xml              |   1 +
>  ...work-source-curl-nbdkit.x86_64-latest.args |  49 ++
>  .../disk-network-source-curl-nbdkit.xml       |   1 +
>  ...isk-network-source-curl.x86_64-latest.args |  53 ++
>  .../disk-network-source-curl.xml              |  71 ++
>  tests/qemuxml2argvtest.c                      |  12 +
>  tests/testutilsqemu.c                         |  16 +
>  tests/testutilsqemu.h                         |   4 +
>  32 files changed, 1302 insertions(+), 53 deletions(-)
>  create mode 100644 src/qemu/qemu_nbdkit.c
>  create mode 100644 src/qemu/qemu_nbdkit.h
>  create mode 100644 tests/qemuxml2argvdata/disk-cdrom-network-nbdkit.x86_64-latest.args
>  create mode 120000 tests/qemuxml2argvdata/disk-cdrom-network-nbdkit.xml
>  create mode 100644 tests/qemuxml2argvdata/disk-network-http-nbdkit.x86_64-latest.args
>  create mode 120000 tests/qemuxml2argvdata/disk-network-http-nbdkit.xml
>  create mode 100644 tests/qemuxml2argvdata/disk-network-source-curl-nbdkit.x86_64-latest.args
>  create mode 120000 tests/qemuxml2argvdata/disk-network-source-curl-nbdkit.xml
>  create mode 100644 tests/qemuxml2argvdata/disk-network-source-curl.x86_64-latest.args
>  create mode 100644 tests/qemuxml2argvdata/disk-network-source-curl.xml
>