[virt-tools-list] ANNOUNCE: libnbd 1.2 & nbdkit 1.16 - high performance NBD client and server

Richard W.M. Jones rjones at redhat.com
Thu Nov 14 11:45:47 UTC 2019


I'm pleased to announce the releases of libnbd 1.2 and nbdkit 1.16.
These are a high performance Network Block Device (NBD) client library
and server.


Key features of libnbd:

 * Synchronous API for ease of use.
 * Asynchronous API for writing non-blocking, multithreaded clients.
   You can mix both APIs freely.
 * High performance.
 * Minimal dependencies for the basic library.
 * Well-documented, stable API.
 * Bindings in several programming languages.
 * Shell (nbdsh) for command line and scripting.

Git: https://github.com/libguestfs/libnbd
Download: http://download.libguestfs.org/libnbd/1.2-stable/
Fedora: https://koji.fedoraproject.org/koji/packageinfo?packageID=28807


Key features of nbdkit:

 * Multithreaded NBD server written in C with good performance.
 * Minimal dependencies for the basic server.
 * Liberal license (BSD) allows nbdkit to be linked to proprietary
   libraries or included in proprietary code.
 * Well-documented, simple plugin API with a stable ABI guarantee.
   Lets you export “unconventional” block devices easily.
 * You can write plugins in C, Lua, Perl, Python, OCaml, Ruby, Rust,
   shell script or Tcl.
 * Filters can be stacked in front of plugins to transform the output.

Git: https://github.com/libguestfs/nbdkit
Download: http://download.libguestfs.org/nbdkit/1.16-stable/
Fedora: https://koji.fedoraproject.org/koji/packageinfo?packageID=16469


*** Release notes for libnbd 1.2 ***

       These are the release notes for libnbd stable release 1.2.  This
       describes the major changes since 1.0.

       libnbd 1.2.0 was released on 14th November 2019.

   Security
       Two security problems were found during development of libnbd 1.2.
       Both were backported to the 1.0 stable branch.  Upgrading is highly
       advisable.

       CVE-2019-14842 protocol downgrade attack when using
       "LIBNBD_TLS_REQUIRE"

       See the full announcement and links to mitigation, tests and fixes
       here:
       https://www.redhat.com/archives/libguestfs/2019-September/msg00128.html

       remote code execution vulnerability

       See the full announcement here:
       https://www.redhat.com/archives/libguestfs/2019-October/msg00060.html

   New APIs
       nbd_can_fast_zero(3)
           Test support by the server for fast zeroing (Eric Blake).

       nbd_connect_socket(3)
       nbd_aio_connect_socket(3)
           Connect to a local connected socket which you create in your main
           program using your own chosen method.

       nbd_connect_systemd_socket_activation(3)
       nbd_aio_connect_systemd_socket_activation(3)
           Connect to local processes that support systemd socket activation.

       nbd_connect_vsock(3)
       nbd_aio_connect_vsock(3)
           Used to connect to servers over "AF_VSOCK".

       nbd_get_handshake_flags(3)
       nbd_set_handshake_flags(3)
       nbd_get_request_structured_replies(3)
       nbd_set_request_structured_replies(3)
       nbd_get_structured_replies_negotiated(3)
           Can be used when testing NBD servers to avoid various NBD features
           (Eric Blake).

       nbd_get_protocol(3)
           Get the NBD protocol variant that the server supports.

       nbd_get_tls_negotiated(3)
           Did we actually negotiate a TLS connection?

       nbd_set_uri_allow_local_file(3)
       nbd_set_uri_allow_tls(3)
       nbd_set_uri_allow_transports(3)
           These can be used to filter NBD URIs before calling
           nbd_connect_uri(3).

   New features
       New tool nbdfuse(1) lets you create a loop-mounted file backed by an
       NBD server without needing root.

       "AF_VSOCK" is now a supported protocol (thanks Stefan Hajnoczi and
       Stefano Garzarella).

       Support for the "FAST_ZERO" flag (Eric Blake).

       Allow disabling certain protocol features, to make it easier to test
       servers (Eric Blake).

       Stack-allocated Variable Length Arrays (VLAs) are now banned throughout
       the library, making the library easier to consume from threads and
       other small stack situations.

       Reproducible builds (Chris Lamb).

       Support for filtering potentially dangerous or undesirable NBD URI
       features.

   Documentation
       Many improvements to the generated manual pages, including:

       ·   Separate "RETURN VALUE" and "ERRORS" sections for each API
           function.

       ·   Example code.

       ·   Relevant links can be added to the "SEE ALSO" section.

       ·   Link to NBD URI specification where relevant, and improve
           documentation around what URIs libnbd supports.

       ·   Document libnbd version number scheme.

       ·   Document limits on export name length, encoding etc.

       New libnbd-security(3) man page listing past security issues and
       remediations (Eric Blake).

   Tools
       nbdsh(1) has a new --base-allocation option which can be used to
       request "base:allocation" metadata context.

       New nbdsh(1) --uri (-u) option to connect to URIs.

   Tests
       You can now fuzz libnbd using either American Fuzzy Lop or clang’s
       libFuzzer.

       Add unit tests for nbdsh(1) (Eric Blake).

       Improved interop testing with various NBD servers and features.

   Other improvements and bug fixes
       nbd_connect_tcp(3) now tries to return the correct errno(3) from the
       underlying connect(2) call when that fails.

       The nbd-protocol.h header file is now shared between libnbd and nbdkit.

       Better fork-safety in "nbd_connect_*" APIs.

       The code was analyzed with Coverity and various problems identified and
       fixed.


*** Release notes for nbdkit 1.16 ***

       These are the release notes for nbdkit stable release 1.16.  This
       describes the major changes since 1.14.

       nbdkit 1.16.0 was released on 14th November 2019.

   Security
       Two security issues were found during development of nbdkit 1.16.
       Fixes for these were backported to older stable branches.  Upgrading to
       the fixed versions is highly recommended.  The new nbdkit-security(1)
       man page contains an up to date list of past security issues.

       CVE-2019-14850 denial of service due to premature opening of back-end
       connection

       See the full announcement and links to mitigation, tests and fixes
       here:
       https://www.redhat.com/archives/libguestfs/2019-September/msg00084.html

       CVE-2019-14851 assertion failure by issuing commands in the wrong order

       This CVE was caused by the fix to the previous issue.

       See the full announcement and links to mitigation, tests and fixes
       here:
       https://www.redhat.com/archives/libguestfs/2019-September/msg00272.html

   New features
       Add support for fast zeroing.  Plugins can expose this using the new
       ".can_fast_zero" method (Eric Blake).

       nbdkit-partitioning-plugin(1) allows use of "mbr-id=default" or
       "type-guid=default" to go back to the default MBR byte or partition
       type GUID.

       New --mask-handshake server flag can be used for testing client feature
       negotiation (Eric Blake).

       The client export name is passed to nbdkit-captive(1) --run parameter
       as $exportname (Eric Blake).

       Captive --run commands which fail (eg. aborting) now cause nbdkit to
       exit with an error instead of errors being silently ignored (Eric
       Blake).

       File descriptors can be passed to password parameters, eg:
       "password=-3" which means that the password should be read from file
       descriptor 3.

       nbdkit can now serve over the "AF_VSOCK" protocol (thanks Stefan
       Hajnoczi).

       New --log=null option discards error messages.

   Plugins
       Python 2 support has been dropped from nbdkit-python-plugin(3) in line
       with Python 2 end of life at the beginning of 2020.  Python ≥ 3.3 is
       required by this plugin.  If you wish to continue to use Python 2 then
       you will need to use nbdkit 1.14.

       New nbdkit-info-plugin(1) which returns various server information back
       to the client.  It can be used for testing server latency amongst other
       things.

       nbdkit-data-plugin(1) now allows you to write "BYTE*N" to get repeated
       bytes (eg. nbdkit data data="0x55*4096").

       nbdkit-ssh-plugin(1) new parameter "compression=true|false" to control
       transport compression.

       nbdkit-vddk-plugin(1) is no longer compiled on non-x86 platforms since
       VMware has only ever shipped VDDK on x86.

       nbdkit-sh-plugin(1) scripts can now see the client exportname and can
       use the "magic_config_key" feature.

   Filters
       New nbdkit-retry-filter(1) which can reopen the plugin transparently on
       certain types of failures (lots of help from Eric Blake).

   API
       Macros "NBDKIT_VERSION_MAJOR", "NBDKIT_VERSION_MINOR",
       "NBDKIT_VERSION_MICRO" expose the compile-time version of nbdkit to
       plugins and filters (Eric Blake).

       Filters (which unlike plugins do not have a public stable API) must now
       exactly match the version of nbdkit when loaded (Eric Blake).

       New ".can_fast_zero" method (Eric Blake).

       New "nbdkit_export_name" server function for reading the export name
       passed by the client.

       New "nbdkit_peer_name" server function to return the client address
       (like getpeername(2)).

       New server functions for safely parsing integers: "nbdkit_parse_int",
       "nbdkit_parse_unsigned", "nbdkit_parse_int8_t", "nbdkit_parse_uint8_t",
       "nbdkit_parse_int16_t", "nbdkit_parse_uint16_t",
       "nbdkit_parse_int32_t", "nbdkit_parse_uint32_t",
       "nbdkit_parse_int64_t", "nbdkit_parse_uint64_t".

   Bug fixes
       ".trim" with FUA flag set now works (Eric Blake).

   Documentation
       The previous release notes have been turned into man pages.

   Tests
       Several tests now optionally use nbdsh(1) instead of qemu-io.

       You can now fuzz nbdkit using either American Fuzzy Lop or clang’s
       libFuzzer.

       Several tests have had sleep times increased to make them more stable
       when run on slow or heavily loaded machines.

   Internals
       Reproducible builds (Chris Lamb).

       Compile code with -Wshadow warning (Eric Blake).

       The internal backend system has been extensively overhauled.  In
       particular this means that we now validate request ranges as requests
       are passed between filters and down to the plugin, making it easier to
       find bugs in filters early (Eric Blake).

       Plugin size and "can_*" flags are cached more aggressively by the
       server (Eric Blake).

       Variable Length Arrays (VLAs) on stack are now banned throughout the
       code.

       The nbd-protocol.h header describing the NBD protocol is now shared
       with libnbd(3).

       Plugin ".unload" method is now called after all worker threads have
       exited, avoiding races at server shutdown.

       Code was audited using Coverity and various problems were fixed.




-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v




More information about the virt-tools-list mailing list