[Libguestfs] [libnbd PATCH v3 5/5] python: Slice structured read callback buffer from original
Eric Blake
eblake at redhat.com
Thu Jun 9 13:34:47 UTC 2022
The Py_BuildValue "y#" format copies data; this is because Python
assumes our C memory can go out of scope, while the user's python
callback can stash off whatever bytearray object it got. But this
copying is inefficient. Now that we already have a Python buffer-like
object in scope for the duration of all structured reads
(pread_structured since commit f8c76c01, aio_pread_structured since
commit 4f15d0e9), we know that the C memory parameter to our
chunk_callback is already (a portion of) the underlying contiguous
memory of an original Python object. If we pass the Python callback a
memoryview object over the correct slice of that memory, then Python
will correctly handle the intricacies of what happens if the callback
saves our memoryview object for later use, all without any data
copying overhead, making use of the pread_structured form much faster.
We basically want to code up the C equivalent to the python expression
'view = memoryview(buf).toreadonly()[start:end]', where start and end
are determined by comparing subbuf/count against the original buf.
But in C, that involves creating PyObject integers to form a PySlice,
then using PyObject_GetItem to apply the slice to a PyMemoryView (I
don't know of any faster tricks; googling does not find any quick
hits). Extract this work into a new utils.c helper function, to avoid
repetition (and in case we figure out a way to do it in fewer lines by
directly manipulating Py_buffer objects).
The testsuite is updated to cover examples of the data persistence
aspect, demonstrating that our stashed slice is still tied to the
returned Python buffer, even though the callback is long since
completed (prior to this patch, stash was an independent bytes copy,
and while it persisted on after the callback, it is frozen in time to
the contents of the buffer at the time the callback was reached). The
fact that data is now shared is is an API change, but one unlikely to
cause problems - it is very unusual for a callback function to try and
stash sub-buffers for later use (neither libnbd nor nbdkit was
utilizing that in their testsuites before this patch).
To demonstrate the speedup now that there is less copying involved per
callback, I tested:
$ export script='
def f(b,o,s,e):
pass
m=1024*1024
size=h.get_size()
h.set_pread_initialize(False)
for i in range(size // m):
buf = h.pread_structured(m, m*i, f)
buf = None
'
$ time ./run nbdkit -U - memory 10G --run 'nbdsh -u "$uri" -c "$script"'
On my machine, this took 16.9s with libnbd 1.12.3, 15.5s pre-patch,
and 3.6s post-patch, for more than 4x speedup in the best case. Much
of the differences boil down to how efficient the Python garbage
collector is able to spot liveness of various buffers, to reuse rather
than allocate new buffers on the Python side; removing the 'buf =
None' line speeds up libnbd 1.12.3 to 4.5s (better reuse of the "y#"
memory in the callback when backed by C memory), 9.0s pre-patch
(copying "y#" from Python memory has scanning effects), and 3.0s
post-patch (zero-copying is always best).
A variation with aio_pread_structured reusing the same nbd.Buffer is
not quite as drastic: 3.9s with libnbd 1.12.3, 4.0s pre-patch, and
3.4s with this patch, but still shows that it is beneficial.
The corresponding diff to generated code is:
| --- python/methods.h.bak 2022-06-09 07:23:51.657260039 -0500
| +++ python/methods.h 2022-06-09 07:23:57.629266878 -0500
| @@ -36,6 +36,7 @@
| extern int nbd_internal_py_init_aio_buffer (PyObject *);
| extern PyObject *nbd_internal_py_get_nbd_buffer_type (void);
| extern PyObject *nbd_internal_py_wrap_errptr (int);
| +extern PyObject *nbd_internal_py_get_subview (PyObject *, const char *, size_t);
|
| static inline struct nbd_handle *
| get_handle (PyObject *obj)
| --- python/methods.c.bak 2022-06-09 07:23:51.652260034 -0500
| +++ python/methods.c 2022-06-09 07:23:57.644266895 -0500
| @@ -73,13 +73,15 @@ chunk_wrapper (void *user_data, const vo
|
| PyGILState_STATE py_save = PyGILState_UNLOCKED;
| PyObject *py_args, *py_ret;
| + PyObject *py_subbuf = NULL;
| PyObject *py_error = NULL;
|
| + py_subbuf = nbd_internal_py_get_subview (data->view, subbuf, count);
| + if (!py_subbuf) { PyErr_PrintEx (0); goto out; }
| py_error = nbd_internal_py_wrap_errptr (*error);
| if (!py_error) { PyErr_PrintEx (0); goto out; }
|
| - py_args = Py_BuildValue ("(y#KIO)", subbuf, (int) count, offset, status,
| - py_error);
| + py_args = Py_BuildValue ("(OKIO)", py_subbuf, offset, status, py_error);
| if (!py_args) { PyErr_PrintEx (0); goto out; }
|
| py_save = PyGILState_Ensure ();
| @@ -106,6 +108,7 @@ chunk_wrapper (void *user_data, const vo
| };
|
| out:
| + Py_XDECREF (py_subbuf);
| if (py_error) {
| PyObject *py_error_ret = PyObject_GetAttrString (py_error, "value");
| *error = PyLong_AsLong (py_error_ret);
| @@ -2231,6 +2234,7 @@ nbd_internal_py_pread_structured (PyObje
| /* Increment refcount since pointer may be saved by libnbd. */
| Py_INCREF (py_chunk_fn);
| chunk_user_data->fn = py_chunk_fn;
| + chunk_user_data->view = nbd_internal_py_get_aio_view (buf, PyBUF_WRITE);
|
| ret = nbd_pread_structured (h, PyByteArray_AS_STRING (buf), count,
| offset_u64, chunk, flags_u32);
| @@ -3116,6 +3120,7 @@ nbd_internal_py_aio_pread_structured (Py
| /* Increment refcount since pointer may be saved by libnbd. */
| Py_INCREF (py_chunk_fn);
| chunk_user_data->fn = py_chunk_fn;
| + chunk_user_data->view = nbd_internal_py_get_aio_view (buf, PyBUF_WRITE);
|
| if (nbd_internal_py_init_aio_buffer (buf) < 0) goto out;
| ret = nbd_aio_pread_structured (h, py_buf->buf, py_buf->len, offset_u64,
---
generator/Python.ml | 18 ++++--
python/t/405-pread-structured.py | 95 +++++++++---------------------
python/t/505-aio-pread-callback.py | 93 ++++++++---------------------
python/utils.c | 34 +++++++++++
4 files changed, 99 insertions(+), 141 deletions(-)
diff --git a/generator/Python.ml b/generator/Python.ml
index 975cab4..c424239 100644
--- a/generator/Python.ml
+++ b/generator/Python.ml
@@ -42,6 +42,7 @@ let
extern int nbd_internal_py_init_aio_buffer (PyObject *);
extern PyObject *nbd_internal_py_get_nbd_buffer_type (void);
extern PyObject *nbd_internal_py_wrap_errptr (int);
+extern PyObject *nbd_internal_py_get_subview (PyObject *, const char *, size_t);
static inline struct nbd_handle *
get_handle (PyObject *obj)
@@ -163,8 +164,8 @@ let
pr " PyObject *py_args, *py_ret;\n";
List.iter (
function
- | CBArrayAndLen (UInt32 n, _) ->
- pr " PyObject *py_%s = NULL;\n" n
+ | CBArrayAndLen (UInt32 n, _)
+ | CBBytesIn (n, _)
| CBMutable (Int n) ->
pr " PyObject *py_%s = NULL;\n" n
| _ -> ()
@@ -181,7 +182,9 @@ let
pr " if (!py_e_%s) { PyErr_PrintEx (0); goto out; }\n" n;
pr " PyList_SET_ITEM (py_%s, i_%s, py_e_%s);\n" n n n;
pr " }\n"
- | CBBytesIn _
+ | CBBytesIn (n, len) ->
+ pr " py_%s = nbd_internal_py_get_subview (data->view, %s, %s);\n" n n len;
+ pr " if (!py_%s) { PyErr_PrintEx (0); goto out; }\n" n
| CBInt _
| CBInt64 _ -> ()
| CBMutable (Int n) ->
@@ -198,7 +201,7 @@ let
List.map (
function
| CBArrayAndLen (UInt32 n, _) -> "O", sprintf "py_%s" n
- | CBBytesIn (n, len) -> "y#", sprintf "%s, (int) %s" n len
+ | CBBytesIn (n, _) -> "O", sprintf "py_%s" n
| CBInt n -> "i", n
| CBInt64 n -> "L", n
| CBMutable (Int n) -> "O", sprintf "py_%s" n
@@ -250,6 +253,8 @@ let
function
| CBArrayAndLen (UInt32 n, _) ->
pr " Py_XDECREF (py_%s);\n" n
+ | CBBytesIn (n, _) ->
+ pr " Py_XDECREF (py_%s);\n" n
| CBMutable (Int n) ->
pr " if (py_%s) {\n" n;
pr " PyObject *py_%s_ret = PyObject_GetAttrString (py_%s, \"value\");\n" n n;
@@ -257,7 +262,6 @@ let
pr " Py_DECREF (py_%s_ret);\n" n;
pr " Py_DECREF (py_%s);\n" n;
pr " }\n"
- | CBBytesIn _
| CBInt _ | CBInt64 _
| CBString _
| CBUInt _ | CBUInt64 _ -> ()
@@ -440,7 +444,9 @@ let
pr " }\n";
pr " /* Increment refcount since pointer may be saved by libnbd. */\n";
pr " Py_INCREF (py_%s_fn);\n" cbname;
- pr " %s_user_data->fn = py_%s_fn;\n" cbname cbname
+ pr " %s_user_data->fn = py_%s_fn;\n" cbname cbname;
+ if cbname = "chunk" then
+ pr " chunk_user_data->view = nbd_internal_py_get_aio_view (buf, PyBUF_WRITE);\n"
| Enum _ -> ()
| Flags (n, _) -> pr " %s_u32 = %s;\n" n n
| Fd _ | Int _ -> ()
diff --git a/python/t/405-pread-structured.py b/python/t/405-pread-structured.py
index afd1e7e..9d3df68 100644
--- a/python/t/405-pread-structured.py
+++ b/python/t/405-pread-structured.py
@@ -1,5 +1,5 @@
# libnbd Python bindings
-# Copyright (C) 2010-2019 Red Hat Inc.
+# Copyright (C) 2010-2022 Red Hat Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
@@ -17,76 +17,19 @@
import nbd
import errno
+import sys
+from array import array
h = nbd.NBD()
h.connect_command(["nbdkit", "-s", "--exit-with-parent", "-v",
"pattern", "size=512"])
-expected = (b'\x00\x00\x00\x00\x00\x00\x00\x00'
- + b'\x00\x00\x00\x00\x00\x00\x00\x08'
- + b'\x00\x00\x00\x00\x00\x00\x00\x10'
- + b'\x00\x00\x00\x00\x00\x00\x00\x18'
- + b'\x00\x00\x00\x00\x00\x00\x00 '
- + b'\x00\x00\x00\x00\x00\x00\x00('
- + b'\x00\x00\x00\x00\x00\x00\x000'
- + b'\x00\x00\x00\x00\x00\x00\x008'
- + b'\x00\x00\x00\x00\x00\x00\x00@'
- + b'\x00\x00\x00\x00\x00\x00\x00H'
- + b'\x00\x00\x00\x00\x00\x00\x00P'
- + b'\x00\x00\x00\x00\x00\x00\x00X'
- + b'\x00\x00\x00\x00\x00\x00\x00`'
- + b'\x00\x00\x00\x00\x00\x00\x00h'
- + b'\x00\x00\x00\x00\x00\x00\x00p'
- + b'\x00\x00\x00\x00\x00\x00\x00x'
- + b'\x00\x00\x00\x00\x00\x00\x00\x80'
- + b'\x00\x00\x00\x00\x00\x00\x00\x88'
- + b'\x00\x00\x00\x00\x00\x00\x00\x90'
- + b'\x00\x00\x00\x00\x00\x00\x00\x98'
- + b'\x00\x00\x00\x00\x00\x00\x00\xa0'
- + b'\x00\x00\x00\x00\x00\x00\x00\xa8'
- + b'\x00\x00\x00\x00\x00\x00\x00\xb0'
- + b'\x00\x00\x00\x00\x00\x00\x00\xb8'
- + b'\x00\x00\x00\x00\x00\x00\x00\xc0'
- + b'\x00\x00\x00\x00\x00\x00\x00\xc8'
- + b'\x00\x00\x00\x00\x00\x00\x00\xd0'
- + b'\x00\x00\x00\x00\x00\x00\x00\xd8'
- + b'\x00\x00\x00\x00\x00\x00\x00\xe0'
- + b'\x00\x00\x00\x00\x00\x00\x00\xe8'
- + b'\x00\x00\x00\x00\x00\x00\x00\xf0'
- + b'\x00\x00\x00\x00\x00\x00\x00\xf8'
- + b'\x00\x00\x00\x00\x00\x00\x01\x00'
- + b'\x00\x00\x00\x00\x00\x00\x01\x08'
- + b'\x00\x00\x00\x00\x00\x00\x01\x10'
- + b'\x00\x00\x00\x00\x00\x00\x01\x18'
- + b'\x00\x00\x00\x00\x00\x00\x01 '
- + b'\x00\x00\x00\x00\x00\x00\x01('
- + b'\x00\x00\x00\x00\x00\x00\x010'
- + b'\x00\x00\x00\x00\x00\x00\x018'
- + b'\x00\x00\x00\x00\x00\x00\x01@'
- + b'\x00\x00\x00\x00\x00\x00\x01H'
- + b'\x00\x00\x00\x00\x00\x00\x01P'
- + b'\x00\x00\x00\x00\x00\x00\x01X'
- + b'\x00\x00\x00\x00\x00\x00\x01`'
- + b'\x00\x00\x00\x00\x00\x00\x01h'
- + b'\x00\x00\x00\x00\x00\x00\x01p'
- + b'\x00\x00\x00\x00\x00\x00\x01x'
- + b'\x00\x00\x00\x00\x00\x00\x01\x80'
- + b'\x00\x00\x00\x00\x00\x00\x01\x88'
- + b'\x00\x00\x00\x00\x00\x00\x01\x90'
- + b'\x00\x00\x00\x00\x00\x00\x01\x98'
- + b'\x00\x00\x00\x00\x00\x00\x01\xa0'
- + b'\x00\x00\x00\x00\x00\x00\x01\xa8'
- + b'\x00\x00\x00\x00\x00\x00\x01\xb0'
- + b'\x00\x00\x00\x00\x00\x00\x01\xb8'
- + b'\x00\x00\x00\x00\x00\x00\x01\xc0'
- + b'\x00\x00\x00\x00\x00\x00\x01\xc8'
- + b'\x00\x00\x00\x00\x00\x00\x01\xd0'
- + b'\x00\x00\x00\x00\x00\x00\x01\xd8'
- + b'\x00\x00\x00\x00\x00\x00\x01\xe0'
- + b'\x00\x00\x00\x00\x00\x00\x01\xe8'
- + b'\x00\x00\x00\x00\x00\x00\x01\xf0'
- + b'\x00\x00\x00\x00\x00\x00\x01\xf8')
-
+# The nbdkit pattern plugin exposes 64-bit numbers in bigendian order
+arr = array('q', range(0, 512, 8))
+if sys.byteorder == 'little':
+ arr.byteswap()
+expected = memoryview(arr).cast('B')
+stash = None
def f(user_data, buf2, offset, s, err):
assert err.value == 0
@@ -94,8 +37,15 @@ def f(user_data, buf2, offset, s, err):
if user_data != 42:
raise ValueError('unexpected user_data')
assert buf2 == expected
+ try:
+ buf2[0] = 1
+ assert False
+ except TypeError:
+ pass
assert offset == 0
assert s == nbd.READ_DATA
+ global stash
+ stash = buf2
buf = h.pread_structured(512, 0, lambda *args: f(42, *args))
@@ -104,6 +54,19 @@ print("%r" % buf)
assert buf == expected
+# The callback can stash its slice; as long as that is live, we can't
+# resize buf but can view changes in buf through the slice
+try:
+ buf.pop()
+ assert False
+except BufferError:
+ pass
+buf[0] ^= 1
+assert buf == stash
+stash = None
+buf.pop()
+
+# Tests of error handling
buf = h.pread_structured(512, 0, lambda *args: f(42, *args),
nbd.CMD_FLAG_DF)
diff --git a/python/t/505-aio-pread-callback.py b/python/t/505-aio-pread-callback.py
index 1773aee..5c7b250 100644
--- a/python/t/505-aio-pread-callback.py
+++ b/python/t/505-aio-pread-callback.py
@@ -1,5 +1,5 @@
# libnbd Python bindings
-# Copyright (C) 2010-2019 Red Hat Inc.
+# Copyright (C) 2010-2022 Red Hat Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
@@ -17,76 +17,19 @@
import nbd
import errno
+import sys
+from array import array
h = nbd.NBD()
h.connect_command(["nbdkit", "-s", "--exit-with-parent", "-v",
"pattern", "size=512"])
-expected = (b'\x00\x00\x00\x00\x00\x00\x00\x00'
- + b'\x00\x00\x00\x00\x00\x00\x00\x08'
- + b'\x00\x00\x00\x00\x00\x00\x00\x10'
- + b'\x00\x00\x00\x00\x00\x00\x00\x18'
- + b'\x00\x00\x00\x00\x00\x00\x00 '
- + b'\x00\x00\x00\x00\x00\x00\x00('
- + b'\x00\x00\x00\x00\x00\x00\x000'
- + b'\x00\x00\x00\x00\x00\x00\x008'
- + b'\x00\x00\x00\x00\x00\x00\x00@'
- + b'\x00\x00\x00\x00\x00\x00\x00H'
- + b'\x00\x00\x00\x00\x00\x00\x00P'
- + b'\x00\x00\x00\x00\x00\x00\x00X'
- + b'\x00\x00\x00\x00\x00\x00\x00`'
- + b'\x00\x00\x00\x00\x00\x00\x00h'
- + b'\x00\x00\x00\x00\x00\x00\x00p'
- + b'\x00\x00\x00\x00\x00\x00\x00x'
- + b'\x00\x00\x00\x00\x00\x00\x00\x80'
- + b'\x00\x00\x00\x00\x00\x00\x00\x88'
- + b'\x00\x00\x00\x00\x00\x00\x00\x90'
- + b'\x00\x00\x00\x00\x00\x00\x00\x98'
- + b'\x00\x00\x00\x00\x00\x00\x00\xa0'
- + b'\x00\x00\x00\x00\x00\x00\x00\xa8'
- + b'\x00\x00\x00\x00\x00\x00\x00\xb0'
- + b'\x00\x00\x00\x00\x00\x00\x00\xb8'
- + b'\x00\x00\x00\x00\x00\x00\x00\xc0'
- + b'\x00\x00\x00\x00\x00\x00\x00\xc8'
- + b'\x00\x00\x00\x00\x00\x00\x00\xd0'
- + b'\x00\x00\x00\x00\x00\x00\x00\xd8'
- + b'\x00\x00\x00\x00\x00\x00\x00\xe0'
- + b'\x00\x00\x00\x00\x00\x00\x00\xe8'
- + b'\x00\x00\x00\x00\x00\x00\x00\xf0'
- + b'\x00\x00\x00\x00\x00\x00\x00\xf8'
- + b'\x00\x00\x00\x00\x00\x00\x01\x00'
- + b'\x00\x00\x00\x00\x00\x00\x01\x08'
- + b'\x00\x00\x00\x00\x00\x00\x01\x10'
- + b'\x00\x00\x00\x00\x00\x00\x01\x18'
- + b'\x00\x00\x00\x00\x00\x00\x01 '
- + b'\x00\x00\x00\x00\x00\x00\x01('
- + b'\x00\x00\x00\x00\x00\x00\x010'
- + b'\x00\x00\x00\x00\x00\x00\x018'
- + b'\x00\x00\x00\x00\x00\x00\x01@'
- + b'\x00\x00\x00\x00\x00\x00\x01H'
- + b'\x00\x00\x00\x00\x00\x00\x01P'
- + b'\x00\x00\x00\x00\x00\x00\x01X'
- + b'\x00\x00\x00\x00\x00\x00\x01`'
- + b'\x00\x00\x00\x00\x00\x00\x01h'
- + b'\x00\x00\x00\x00\x00\x00\x01p'
- + b'\x00\x00\x00\x00\x00\x00\x01x'
- + b'\x00\x00\x00\x00\x00\x00\x01\x80'
- + b'\x00\x00\x00\x00\x00\x00\x01\x88'
- + b'\x00\x00\x00\x00\x00\x00\x01\x90'
- + b'\x00\x00\x00\x00\x00\x00\x01\x98'
- + b'\x00\x00\x00\x00\x00\x00\x01\xa0'
- + b'\x00\x00\x00\x00\x00\x00\x01\xa8'
- + b'\x00\x00\x00\x00\x00\x00\x01\xb0'
- + b'\x00\x00\x00\x00\x00\x00\x01\xb8'
- + b'\x00\x00\x00\x00\x00\x00\x01\xc0'
- + b'\x00\x00\x00\x00\x00\x00\x01\xc8'
- + b'\x00\x00\x00\x00\x00\x00\x01\xd0'
- + b'\x00\x00\x00\x00\x00\x00\x01\xd8'
- + b'\x00\x00\x00\x00\x00\x00\x01\xe0'
- + b'\x00\x00\x00\x00\x00\x00\x01\xe8'
- + b'\x00\x00\x00\x00\x00\x00\x01\xf0'
- + b'\x00\x00\x00\x00\x00\x00\x01\xf8')
-
+# The nbdkit pattern plugin exposes 64-bit numbers in bigendian order
+arr = array('q', range(0, 512, 8))
+if sys.byteorder == 'little':
+ arr.byteswap()
+expected = memoryview(arr).cast('B')
+stash = None
def chunk(user_data, buf2, offset, s, err):
print("in chunk, user_data %d" % user_data)
@@ -97,6 +40,8 @@ def chunk(user_data, buf2, offset, s, err):
assert buf2 == expected
assert offset == 0
assert s == nbd.READ_DATA
+ global stash
+ stash = buf2
def callback(user_data, err):
@@ -111,19 +56,29 @@ def callback(user_data, err):
# First try: succeed in both callbacks
-buf = nbd.Buffer(512)
+buf = bytearray(512)
cookie = h.aio_pread_structured(buf, 0,
lambda *args: chunk(42, *args),
lambda *args: callback((42, 42), *args))
while not h.aio_command_completed(cookie):
h.poll(-1)
-buf = buf.to_bytearray()
-
print("%r" % buf)
assert buf == expected
+# The callback can stash its slice; as long as that is live, we can't
+# resize buf but can view changes in buf through the slice
+try:
+ buf.pop()
+ assert False
+except BufferError:
+ pass
+buf[0] ^= 1
+assert buf == stash
+stash = None
+buf.pop()
+
# Second try: fail only during callback
buf = nbd.Buffer(512)
cookie = h.aio_pread_structured(buf, 0,
diff --git a/python/utils.c b/python/utils.c
index cd44b90..d5851f7 100644
--- a/python/utils.c
+++ b/python/utils.c
@@ -192,3 +192,37 @@ nbd_internal_py_wrap_errptr (int err)
return PyObject_CallMethod (py_ctypes_mod, "c_int", "i", err);
}
+
+/* Helper to compute view.toreadonly()[start:end] in chunk callback */
+PyObject *
+nbd_internal_py_get_subview (PyObject *view, const char *subbuf, size_t count)
+{
+ Py_buffer *orig;
+ const char *base;
+ PyObject *start, *end, *slice;
+ PyObject *ret;
+
+ assert (PyMemoryView_Check (view));
+ orig = PyMemoryView_GET_BUFFER (view);
+ assert (PyBuffer_IsContiguous (orig, 'A'));
+ base = orig->buf;
+ assert (subbuf >= base && count <= orig->len &&
+ subbuf + count <= base + orig->len);
+ start = PyLong_FromLong (subbuf - base);
+ if (!start) return NULL;
+ end = PyLong_FromLong (subbuf - base + count);
+ if (!end) { Py_DECREF (start); return NULL; }
+ slice = PySlice_New (start, end, NULL);
+ Py_DECREF (start);
+ Py_DECREF (end);
+ if (!slice) return NULL;
+ ret = PyObject_GetItem (view, slice);
+ Py_DECREF (slice);
+ /* memoryview.toreadonly() was only added in Python 3.8.
+ * PyMemoryView_GetContiguous (ret, PyBuf_READ, 'A') doesn't force readonly.
+ * So we mess around directly with the Py_buffer.
+ */
+ if (ret)
+ PyMemoryView_GET_BUFFER (ret)->readonly = 1;
+ return ret;
+}
--
2.36.1
More information about the Libguestfs
mailing list