<div dir="ltr">Our cbdkit was branched from 1.1.12 with some patches from 1.1.13 added as well. The project has morphed significantly enough that a direct diff or merging between the two would not be feasible. Even the structure of the project directory and build has been changed to be in line with our other internal projects.<div><br></div><div>I have uploaded the entire cbdkit source to our github at</div><div><a href="https://github.com/dev-cloudbd/cbdkit">https://github.com/dev-cloudbd/cbdkit</a><br></div><div><br></div><div>The relevant files are</div><div> include/cbdkit-plugin.h</div><div> src/connections.c</div><div> src/plugins.c</div><div><br></div><div>Specifically, the connections.c functions</div><div> recv_request</div><div> send_reply</div><div><br></div><div> and the plugins.c functions</div><div> plugin_pread</div><div> plugin_pwrite</div><div> cbdkit_async_reply</div><div> cbdkit_async_reply_read</div><div> cbdkit_async_reply_error</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jan 19, 2018 at 12:05 PM, Eric Blake <span dir="ltr"><<a href="mailto:eblake@redhat.com" target="_blank">eblake@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On 01/19/2018 10:56 AM, Shaun McDowell wrote:<br>
<br>
> Limitation: The kernel will (with today's default settings) typically be<br>
> willing to send up to 128 requests of 128kB size to the driver in parallel.<br>
> We wanted to support 128 parallel read operations on different areas of the<br>
> disk without requiring 128 separate threads and connections for the driver.<br>
> Right now in nbdkit that is impossible. The main loop in connection.c will<br>
> pull an nbd request off the socket and block until that read request is<br>
> complete before sending a response and getting the next request, blocking<br>
> other requests on the socket unless running X connections/threads in<br>
> parallel.<br>
<br>
</span>What version nbdkit are you using? We recently added parallel reads in<br>
1.1.17 (although some minor fixes went in later; current version is<br>
1.1.25) that should allow you to have a single socket serving multiple<br>
requests in parallel, in response to your setting of nbdkit's --thread<br>
option, and if your plugin is truly parallel (nbdkit now ships both a<br>
'file' and 'nbd' plugin that are truly parallel).<br>
<span class=""><br>
> Change: We introduced an additional set of functions to the nbdkit_plugin<br>
> struct that supports asynchronous handling of the requests and a few helper<br>
> functions for the plugin to use to respond when it has finished the<br>
> request. This is very similar to the fuse filesystem low level api (async<br>
> supported) vs the high level fuse fs api (sync only). The design goal here<br>
> is that a single connection/thread on nbdkit can support as many requests<br>
> in parallel as the plugin allows. The nbdkit side pulls the request off the<br>
> socket and if the async function pointer is non-null it will wrap the<br>
> request in an op struct and use the async plugin call for read/write/etc<br>
> capturing any buffer allocated and some op details into the op pointer. The<br>
> plugin async_* will start the op and return to nbdkit while the plugin<br>
> works on it in the background. Nbdkit will then go back to the socket and<br>
> begin the next request. Our plugin uses 1 connection/nbdkit thread and 2-4<br>
> threads internally with boost asio over sockets to service the requests to<br>
> cloud. We are able to achieve ~1GB/s (yes bytes) read/write performance to<br>
> aws s3 from an ec2 node with 10 gigabit networking on < 100MB of memory in<br>
> the driver with this approach.<br>
<br>
</span>Definitely post patches to the list! My work to add parallel support<br>
via --threads still spawns multiple threads (the plugin is operating<br>
concurrently on multiple threads) while yours is a different approach of<br>
breaking things into smaller stages that piece together and possible<br>
with fewer threads.<br>
<div><div class="h5"><br>
><br>
> Here are some of what our function prototypes look like that support an<br>
> asynchronous nbdkit model<br>
><br>
> #define CBDKIT_THREAD_MODEL_SERIALIZE_<wbr>REQUESTS 2<br>
> #define CBDKIT_THREAD_MODEL_PARALLEL 3<br>
> #define CBDKIT_THREAD_MODEL_ASYNC 4<br>
><br>
> struct cbdkit_plugin {<br>
> ...<br>
> int (*pread) (void *handle, void *buf, uint32_t count, uint64_t offset);<br>
> int (*pwrite) (void *handle, const void *buf, uint32_t count, uint64_t<br>
> offset);<br>
> int (*flush) (void *handle);<br>
> int (*trim) (void *handle, uint32_t count, uint64_t offset);<br>
> int (*zero) (void *handle, uint32_t count, uint64_t offset, int may_trim);<br>
><br>
> int errno_is_preserved;<br>
><br>
> void (*async_pread) (void *op, void *handle, void *buf, uint32_t count,<br>
> uint64_t offset);<br>
> void (*async_pwrite) (void *op, void *handle, const void *buf, uint32_t<br>
> count, uint64_t offset, int fua);<br>
> void (*async_flush) (void *op, void *handle);<br>
> void (*async_trim) (void *op, void *handle, uint32_t count, uint64_t<br>
> offset, int fua);<br>
> void (*async_zero) (void *op, void *handle, uint32_t count, uint64_t<br>
> offset, int may_trim, int fua);<br>
> ...<br>
> }<br>
><br>
> Additionally there are a few helper functions for the plugin to use to<br>
> respond back to nbdkit when the job is eventually finished. The plugin<br>
> contract when using the async functions is that every async func guarantees<br>
> it will call an appropriate async_reply function.<br>
><br>
> /* call for completion of successful async_pwrite, async_flush,<br>
> async_trim, or async_zero */<br>
> extern CBDKIT_CXX_LANG_C int cbdkit_async_reply (void *op);<br>
> /* call for complete of successful async_pread */<br>
> extern CBDKIT_CXX_LANG_C int cbdkit_async_reply_read (void *op);<br>
> /* call for completion of any async operation with error */<br>
> extern CBDKIT_CXX_LANG_C int cbdkit_async_reply_error (void *op, uint32_t<br>
> error);<br>
><br>
> If there is any interest in supporting async ops in the next api version I<br>
> am able to share the entire modified nbdkit (cbdkit) source that we use<br>
> that supports this async op framework, fua, as well as some buffer pooling.<br>
<br>
</div></div>Yes, please post patches.<br>
<div class="HOEnZb"><div class="h5"><br>
<br>
--<br>
Eric Blake, Principal Software Engineer<br>
Red Hat, Inc. <a href="tel:%2B1-919-301-3266" value="+19193013266">+1-919-301-3266</a><br>
Virtualization: <a href="http://qemu.org" rel="noreferrer" target="_blank">qemu.org</a> | <a href="http://libvirt.org" rel="noreferrer" target="_blank">libvirt.org</a><br>
<br>
</div></div></blockquote></div><br></div>