[Libguestfs] [PATCH libnbd v2] golang: examples: Add simple_copy and aio_copy examples
Nir Soffer
nsoffer at redhat.com
Wed Jan 26 21:08:33 UTC 2022
Show how to read entire image using the simple synchronous API and how
and the high performance asynchronous API.
For simplicity, the example do not use extents, do not sparsify the
image, and copy the image to only to stdout. This make it easy to
evaluate the Go bindings performance.
The aio_copy example includes an interesting ordering queue, ensuring
that asynchronous reads are written in the right order even if they
completed out of order. This allows using the fast asynchronous API even
when the output does not support seek, and may perform better even if
the output does support seek when using rotational disks.
The aio_copy example does not use AioBuffer in the normal way to avoid
unwanted copy when AioPread completes. Instead, we use a Go allocated
buffer, in the same way we pass a Go allocated buffer to libnbd in the
synchronous API. This usage is unsafe but required for getting decent
performance.
Testing show that we get better performance than nbdcopy with the
defaults, or the same performance if we tune nbdcopy to use the one
connection and 4 requests, and drop the data using null: output.
All tests use 6 GiB fully allocated zero image created with:
$ dd if=/dev/zero bs=1M count=6144 of=zero-6g.raw
The image is served using qemu-nbd:
$ qemu-nbd -r -t -e0 -k /tmp/nbd.sock --cache=none --aio=native -f raw zero-6g.raw
$ hyperfine "simple_copy/simple_copy $URL >/dev/null" \
"nbdcopy --synchronous --allocated $URL /dev/null" \
"nbdcopy --request-size $((2048*1024)) --synchronous --allocated $URL /dev/null"
Benchmark 1: simple_copy/simple_copy nbd+unix:///?socket=/tmp/nbd.sock >/dev/null
Time (mean ± σ): 3.210 s ± 0.065 s [User: 0.275 s, System: 0.836 s]
Range (min … max): 3.117 s … 3.298 s 10 runs
Benchmark 2: nbdcopy --synchronous --allocated nbd+unix:///?socket=/tmp/nbd.sock /dev/null
Time (mean ± σ): 4.469 s ± 0.019 s [User: 0.295 s, System: 0.948 s]
Range (min … max): 4.447 s … 4.510 s 10 runs
Benchmark 3: nbdcopy --request-size 2097152 --synchronous --allocated nbd+unix:///?socket=/tmp/nbd.sock /dev/null
Time (mean ± σ): 3.266 s ± 0.012 s [User: 0.216 s, System: 0.732 s]
Range (min … max): 3.244 s … 3.286 s 10 runs
Summary
'simple_copy/simple_copy nbd+unix:///?socket=/tmp/nbd.sock >/dev/null' ran
1.02 ± 0.02 times faster than 'nbdcopy --request-size 2097152 --synchronous --allocated nbd+unix:///?socket=/tmp/nbd.sock /dev/null'
1.39 ± 0.03 times faster than 'nbdcopy --synchronous --allocated nbd+unix:///?socket=/tmp/nbd.sock /dev/null'
$ hyperfine "aio_copy/aio_copy $URL >/dev/null" \
"nbdcopy --allocated $URL /dev/null" \
"nbdcopy --allocated $URL null:" \
"nbdcopy --connections 1 --requests 4 --allocated $URL null:"
Benchmark 1: aio_copy/aio_copy nbd+unix:///?socket=/tmp/nbd.sock >/dev/null
Time (mean ± σ): 2.013 s ± 0.035 s [User: 0.410 s, System: 0.877 s]
Range (min … max): 1.966 s … 2.060 s 10 runs
Benchmark 2: nbdcopy --allocated nbd+unix:///?socket=/tmp/nbd.sock /dev/null
Time (mean ± σ): 4.501 s ± 0.025 s [User: 0.287 s, System: 0.949 s]
Range (min … max): 4.449 s … 4.532 s 10 runs
Benchmark 3: nbdcopy --allocated nbd+unix:///?socket=/tmp/nbd.sock null:
Time (mean ± σ): 2.422 s ± 0.018 s [User: 0.520 s, System: 1.772 s]
Range (min … max): 2.404 s … 2.470 s 10 runs
Benchmark 4: nbdcopy --connections 1 --requests 4 --allocated nbd+unix:///?socket=/tmp/nbd.sock null:
Time (mean ± σ): 2.019 s ± 0.009 s [User: 0.270 s, System: 0.845 s]
Range (min … max): 2.008 s … 2.033 s 10 runs
Summary
'aio_copy/aio_copy nbd+unix:///?socket=/tmp/nbd.sock >/dev/null' ran
1.00 ± 0.02 times faster than 'nbdcopy --connections 1 --requests 4 --allocated nbd+unix:///?socket=/tmp/nbd.sock null:'
1.20 ± 0.02 times faster than 'nbdcopy --allocated nbd+unix:///?socket=/tmp/nbd.sock null:'
2.24 ± 0.04 times faster than 'nbdcopy --allocated nbd+unix:///?socket=/tmp/nbd.sock /dev/null'
Signed-off-by: Nir Soffer <nsoffer at redhat.com>
---
Chagnes in v2:
- Fix error handling: panic if aio_pread completion callback *error is
non-zero.
- Create AioPreadOptargs struct instead of pointer. This may save
unneeded allocation per read and is little bit simpler.
V1 was here:
https://listman.redhat.com/archives/libguestfs/2022-January/msg00165.html
golang/examples/Makefile.am | 26 ++-
golang/examples/aio_copy/aio_copy.go | 198 +++++++++++++++++++++
golang/examples/aio_copy/go.mod | 4 +
golang/examples/simple_copy/go.mod | 4 +
golang/examples/simple_copy/simple_copy.go | 93 ++++++++++
5 files changed, 323 insertions(+), 2 deletions(-)
create mode 100644 golang/examples/aio_copy/aio_copy.go
create mode 100644 golang/examples/aio_copy/go.mod
create mode 100644 golang/examples/simple_copy/go.mod
create mode 100644 golang/examples/simple_copy/simple_copy.go
diff --git a/golang/examples/Makefile.am b/golang/examples/Makefile.am
index bfeee245..90f9fbf8 100644
--- a/golang/examples/Makefile.am
+++ b/golang/examples/Makefile.am
@@ -16,27 +16,49 @@
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
include $(top_srcdir)/subdir-rules.mk
EXTRA_DIST = \
LICENSE-FOR-EXAMPLES \
get_size/go.mod \
get_size/get_size.go \
read_first_sector/go.mod \
read_first_sector/read_first_sector.go \
+ simple_copy/go.mod \
+ simple_copy/simple_copy.go \
+ aio_copy/go.mod \
+ aio_copy/aio_copy.go \
$(NULL)
if HAVE_GOLANG
-noinst_SCRIPTS = get_size/get_size read_first_sector/read_first_sector
+noinst_SCRIPTS = \
+ get_size/get_size \
+ read_first_sector/read_first_sector \
+ simple_copy/simple_copy \
+ aio_copy/aio_copy \
+ $(NULL)
get_size/get_size: get_size/get_size.go
cd get_size && \
$(abs_top_builddir)/run go build -o get_size
read_first_sector/read_first_sector: read_first_sector/read_first_sector.go
cd read_first_sector && \
$(abs_top_builddir)/run go build -o read_first_sector
+simple_copy/simple_copy: simple_copy/simple_copy.go
+ cd simple_copy && \
+ $(abs_top_builddir)/run go build -o simple_copy
+
+aio_copy/aio_copy: aio_copy/aio_copy.go
+ cd aio_copy && \
+ $(abs_top_builddir)/run go build -o aio_copy
+
endif HAVE_GOLANG
-CLEANFILES += get_size/get_size read_first_sector/read_first_sector
+CLEANFILES += \
+ get_size/get_size \
+ read_first_sector/read_first_sector \
+ simple_copy/simple_copy \
+ aio_copy/aio_copy \
+ $(NULL)
diff --git a/golang/examples/aio_copy/aio_copy.go b/golang/examples/aio_copy/aio_copy.go
new file mode 100644
index 00000000..b6f5def1
--- /dev/null
+++ b/golang/examples/aio_copy/aio_copy.go
@@ -0,0 +1,198 @@
+/* libnbd example
+ * Copyright (C) 2013-2022 Red Hat Inc.
+ * Examples are under a permissive BSD-like license. See also
+ * golang/examples/LICENSE-For-EXAMPLES
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are
+ * met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * * Neither the name of Red Hat nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+ * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+ * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+// Copy image from NBD URI to stdout.
+//
+// Example:
+//
+// ./aio_copy nbd+unix:///?socket=/tmp.nbd >/dev/null
+//
+package main
+
+import (
+ "container/list"
+ "flag"
+ "os"
+ "sync"
+ "syscall"
+ "unsafe"
+
+ "libguestfs.org/libnbd"
+)
+
+var (
+ // These options give best performance with fast NVMe drive.
+ requestSize = flag.Uint("request-size", 256*1024, "maximum request size in bytes")
+ requests = flag.Uint("requests", 4, "maximum number of requests in flight")
+
+ h *libnbd.Libnbd
+
+ // Keeping commands in a queue ensures commands are written in the right
+ // order, even if they complete out of order. This allows parallel reads
+ // with non-seekable output.
+ queue list.List
+
+ // Buffer pool allocating buffers as needed and reusing them.
+ bufPool = sync.Pool{
+ New: func() interface{} {
+ return make([]byte, *requestSize)
+ },
+ }
+)
+
+// command keeps state of single AioPread call while the read is handled by
+// libnbd, until the command reach the front of the queue and can be writen to
+// the output.
+type command struct {
+ buf []byte
+ length uint
+ ready bool
+}
+
+func main() {
+ flag.Parse()
+
+ var err error
+
+ h, err = libnbd.Create()
+ if err != nil {
+ panic(err)
+ }
+ defer h.Close()
+
+ err = h.ConnectUri(flag.Arg(0))
+ if err != nil {
+ panic(err)
+ }
+
+ size, err := h.GetSize()
+ if err != nil {
+ panic(err)
+ }
+
+ var offset uint64
+
+ for offset < size || queue.Len() > 0 {
+
+ for offset < size && inflightRequests() < *requests {
+ length := *requestSize
+ if size-offset < uint64(length) {
+ length = uint(size - offset)
+ }
+ startRead(offset, length)
+ offset += uint64(length)
+ }
+
+ waitForCompletion()
+
+ for readReady() {
+ finishRead()
+ }
+ }
+}
+
+func inflightRequests() uint {
+ n, err := h.AioInFlight()
+ if err != nil {
+ panic(err)
+ }
+ return n
+}
+
+func waitForCompletion() {
+ start := inflightRequests()
+
+ for {
+ _, err := h.Poll(-1)
+ if err != nil {
+ panic(err)
+ }
+
+ if inflightRequests() < start {
+ break // A read completed.
+ }
+ }
+}
+
+func startRead(offset uint64, length uint) {
+ buf := bufPool.Get().([]byte)
+
+ // Keep buffer in command so we can put it back into the pool when the
+ // command completes.
+ cmd := &command{buf: buf, length: length}
+
+ // Create aio buffer from pool buffer to avoid unneeded allocation for
+ // every read, and unneeded copy when completing the read.
+ abuf := libnbd.AioBuffer{P: unsafe.Pointer(&buf[0]), Size: length}
+
+ args := libnbd.AioPreadOptargs{
+ CompletionCallbackSet: true,
+ CompletionCallback: func(error *int) int {
+ if *error != 0 {
+ // This is not documented, but *error is errno value translated
+ // from the the NBD server error.
+ err := syscall.Errno(*error).Error()
+ panic(err)
+ }
+ cmd.ready = true
+ return 1
+ },
+ }
+
+ _, err := h.AioPread(abuf, offset, &args)
+ if err != nil {
+ panic(err)
+ }
+
+ queue.PushBack(cmd)
+}
+
+func readReady() bool {
+ return queue.Len() > 0 && queue.Front().Value.(*command).ready
+}
+
+func finishRead() {
+ e := queue.Front()
+ queue.Remove(e)
+
+ cmd := e.Value.(*command)
+ b := cmd.buf[:cmd.length]
+
+ _, err := os.Stdout.Write(b)
+ if err != nil {
+ panic(err)
+ }
+
+ bufPool.Put(cmd.buf)
+}
diff --git a/golang/examples/aio_copy/go.mod b/golang/examples/aio_copy/go.mod
new file mode 100644
index 00000000..074fabf7
--- /dev/null
+++ b/golang/examples/aio_copy/go.mod
@@ -0,0 +1,4 @@
+module main
+
+replace libguestfs.org/libnbd => ../../
+require libguestfs.org/libnbd v1.11.5
diff --git a/golang/examples/simple_copy/go.mod b/golang/examples/simple_copy/go.mod
new file mode 100644
index 00000000..074fabf7
--- /dev/null
+++ b/golang/examples/simple_copy/go.mod
@@ -0,0 +1,4 @@
+module main
+
+replace libguestfs.org/libnbd => ../../
+require libguestfs.org/libnbd v1.11.5
diff --git a/golang/examples/simple_copy/simple_copy.go b/golang/examples/simple_copy/simple_copy.go
new file mode 100644
index 00000000..e8fa1f76
--- /dev/null
+++ b/golang/examples/simple_copy/simple_copy.go
@@ -0,0 +1,93 @@
+/* libnbd example
+ * Copyright (C) 2013-2022 Red Hat Inc.
+ * Examples are under a permissive BSD-like license. See also
+ * golang/examples/LICENSE-For-EXAMPLES
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are
+ * met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * * Neither the name of Red Hat nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+ * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+ * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+// Copy image from NBD URI to stdout.
+//
+// Example:
+//
+// ./simple_copy nbd+unix:///?socket=/tmp.nbd >/dev/null
+//
+package main
+
+import (
+ "flag"
+ "os"
+
+ "libguestfs.org/libnbd"
+)
+
+var (
+ requestSize = flag.Uint("buffer-size", 2048*1024, "maximum request size in bytes")
+)
+
+func main() {
+ flag.Parse()
+
+ h, err := libnbd.Create()
+ if err != nil {
+ panic(err)
+ }
+ defer h.Close()
+
+ err = h.ConnectUri(flag.Arg(0))
+ if err != nil {
+ panic(err)
+ }
+
+ size, err := h.GetSize()
+ if err != nil {
+ panic(err)
+ }
+
+ buf := make([]byte, *requestSize)
+ var offset uint64
+
+ for offset < size {
+ if size-offset < uint64(len(buf)) {
+ buf = buf[:offset-size]
+ }
+
+ err = h.Pread(buf, offset, nil)
+ if err != nil {
+ panic(err)
+ }
+
+ _, err := os.Stdout.Write(buf)
+ if err != nil {
+ panic(err)
+ }
+
+ offset += uint64(len(buf))
+ }
+}
--
2.34.1
More information about the Libguestfs
mailing list