[Libguestfs] How to speed up Kernel Client - S3 plugin use-case

Nikolaus Rath Nikolaus at rath.org
Mon Jun 13 09:33:58 UTC 2022


I am trying to improve performance of the scenario where the kernel's
NBD client talks to NBDKit's S3 plugin.

For me, the main bottleneck is currently due to the fact that the kernel
aligns requests to only 512 B, no matter the blocksize reported by

Using a 512 B object size is not feasible (due to latency and request
overhead). However, with a larger object size there are two conflicting

1. To maximize parallelism (which is important to reduce the effects of
connection latency), it's best to limit the size of the kernel's NBD
requests to the object size.

2. To minimize un-aligned writes, it's best to allow arbitrarily large
NBD requests, because the larger the requests the larger the amount of
full blocks that are written. Unfortunately this means that all objects
touched by the request are written sequentially.

I see a number of ways to address that:

1. Change the kernel's NBD code to honor the blocksize reported by the
   NBD server. This would be ideal, but I don't feel up to making this
   happen. Theoretical solution only.

2. Change the S3 plugin to use multiple threads, so that it can upload
   multiple objects in parallel even when they're part of the same NBD
   request. The disadvantage is that this adds a second "layer" of
   threads, in addition to those started by NBDkit itself.

3. Change NBDkit itself to split up requests *and* distribute them to
   multiple threads. I believe this means changes to the core code
   because the blocksize filter can't dispatch requests to multiple

What do people think is the best way to proceed? Is there a fourth
option that I might be missing?


GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«

More information about the Libguestfs mailing list