<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Dec 16, 2020 at 11:36 PM Marc-André Lureau <<a href="mailto:marcandre.lureau@gmail.com">marcandre.lureau@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">Hi<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Dec 15, 2020 at 8:22 PM Jiachen Zhang <<a href="mailto:zhangjiachen.jaycee@bytedance.com" target="_blank">zhangjiachen.jaycee@bytedance.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi, all<br>
<br>
We implement virtio-fs crash reconnection in this patchset. The crash<br>
reconnection of virtiofsd here is completely transparent to guest, no<br>
remount in guest is needed, even the inflight requests can be handled<br>
normally after reconnection. We are looking forward to any comments. <br>
<br>
Thanks,<br>
Jiachen<br>
<br>
<br>
OVERVIEW:<br>
<br>
To support virtio-fs crash reconnection, we need to support the recovery<br>
of 1) inflight FUSE request, and 2) virtiofsd internal status information.<br>
<br>
Fortunately, QEMU's vhost-user reconnection framework already supports<br>
inflight I/O tracking by using VHOST_USER_GET_INFLIGHT_FD and<br>
VHOST_USER_SET_INFLIGHT_FD (see 5ad204bf2 and 5f9ff1eff for details).<br>
As the FUSE requests are transferred by virtqueue I/O requests, by using<br>
the vhost-user inflight I/O tracking, we can recover the inflight FUSE<br>
requests.<br>
<br>
To support virtiofsd internal status recovery, we introduce 4 new<br>
vhost-user message types. As shown in the following diagram, two of them<br>
are used to persist shared lo_maps and opened fds to QEMU, the other two<br>
message types are used to restore the status when reconnecting.<br>
<br>
VHOST_USER_SLAVE_SHM<br>
VHOST_USER_SLAVE_FD<br>
+--------------+ Persist +--------------------+<br>
| <---------------------+ |<br>
| QEMU | | Virtio-fs Daemon |<br>
| +---------------------> |<br>
+--------------+ Restore +--------------------+<br>
VHOST_USER_SET_SHM<br>
VHOST_USER_SET_FD<br>
<br>
Although the 4 newly added message types are to support virtiofsd<br>
reconnection in this patchset, it might be potential in other situation.<br>
So we keep in mind to make them more general when add them to vhost<br>
related source files. VHOST_USER_SLAVE_SHM and VHOST_USER_SET_SHM can be<br>
used for memory sharing between a vhost-user daemon and QEMU,<br>
VHOST_USER_SLAVE_FD and VHOST_USER_SET_FD would be useful if we want to<br>
shared opened fds between QEMU process and vhost-user daemon process.<br></blockquote><div><br></div><div>Before adding new messages to the already complex vhost-user protocol, can we evaluate other options?</div><div><br></div><div>First thing that came to my mind is that the memory state could be saved to disk or with a POSIX shared memory object.</div><div> <br></div></div></div></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div></div><div>Eventually, the protocol could just pass around the fds, and not make a special treatment for shared memory.</div><div><br></div><div>Then I remember systemd has a pretty good API & protocol for this sort of thing: sd_notify(3) (afaik, it is quite easy to implement a minimal handler)</div><div><br></div><div>You can store fds with <span>FDSTORE=1 (with an optional associated <span>FDNAME). sd_listen_fds() & others to get them back (note: passed by inheritance only I think). systemd seems to not make shm a special case either, just treat it like an opened fd to restore. <br></span></span></div><div><span><span><br></span></span></div><div>If we consider backend processes are going to be managed by libvirt or even a systemd service, is it a better alternative? sd_notify() offers a number of interesting features as well to monitor services.<br></div><div><br></div></div></div></blockquote><div><br></div><div><br></div><div><div>Thanks for the suggestions. Actually, we choose to save all state information to QEMU because a virtiofsd has the same lifecycle as its QEMU master. However, saving things to a file do avoid communication with QEMU, and we no longer need to increase the complexity of vhost-user protocol. The suggestion to save fds to the systemd is also very reasonable if we don't consider the lifecycle issues, we will try it.</div></div><div><br></div><div>All the best,</div><div>Jiachen</div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
<br>
USAGE and NOTES:<br>
<br>
- The commits are rebased to a recent QEMU master commit b4d939133dca0fa2b.<br>
<br>
- ",reconnect=1" should be added to the "-chardev socket" of vhost-user-fs-pci<br>
in the QEMU command line, for example:<br>
<br>
qemu-system-x86_64 ... \<br>
-chardev socket,id=char0,path=/tmp/vhostqemu,reconnect=1 \<br>
-device vhost-user-fs-pci,queue-size=1024,chardev=char0,tag=myfs \<br>
...<br>
<br>
- We add new options for virtiofsd to enable or disable crash reconnection.<br>
And some options are not supported by crash reconnection. So add following<br>
options to virtiofsd to enable reconnection:<br>
<br>
virtiofsd ... -o reconnect -o no_mount_ns -o no_flock -o no_posix_lock<br>
-o no_xattr ...<br>
<br>
- The reasons why virtiofsd-side locking, extended attributes, and mount<br>
namespace are not supported is explained in the commit message of the 6th<br>
patch (virtiofsd: Add two new options for crash reconnection).<br>
<br>
- The 9th patch is a work-around that will not affect the overall correctness.<br>
We remove the qsort related codes because we found that when resubmit_num is<br>
larger than 64, seccomp will kill the virtiofsd process.<br>
<br>
- Support for dax version virtiofsd is very possible and requires almost no<br>
additional change to this patchset.<br>
<br>
<br>
Jiachen Zhang (9):<br>
vhost-user-fs: Add support for reconnection of vhost-user-fs backend<br>
vhost: Add vhost-user message types for sending shared memory and file<br>
fds<br>
vhost-user-fs: Support virtiofsd crash reconnection<br>
libvhost-user: Add vhost-user message types for sending shared memory<br>
and file fds<br>
virtiofsd: Convert the struct lo_map array to a more flatten layout<br>
virtiofsd: Add two new options for crash reconnection<br>
virtiofsd: Persist/restore lo_map and opened fds to/from QEMU<br>
virtiofsd: Ensure crash consistency after reconnection<br>
virtiofsd: (work around) Comment qsort in inflight I/O tracking<br>
<br>
contrib/libvhost-user/libvhost-user.c | 106 +++-<br>
contrib/libvhost-user/libvhost-user.h | 70 +++<br>
docs/interop/vhost-user.rst | 41 ++<br>
hw/virtio/vhost-user-fs.c | 334 ++++++++++-<br>
hw/virtio/vhost-user.c | 123 ++++<br>
hw/virtio/vhost.c | 42 ++<br>
include/hw/virtio/vhost-backend.h | 6 +<br>
include/hw/virtio/vhost-user-fs.h | 16 +-<br>
include/hw/virtio/vhost.h | 42 ++<br>
tools/virtiofsd/fuse_lowlevel.c | 24 +-<br>
tools/virtiofsd/fuse_virtio.c | 44 ++<br>
tools/virtiofsd/fuse_virtio.h | 1 +<br>
tools/virtiofsd/helper.c | 9 +<br>
tools/virtiofsd/passthrough_helpers.h | 2 +-<br>
tools/virtiofsd/passthrough_ll.c | 830 ++++++++++++++++++--------<br>
tools/virtiofsd/passthrough_seccomp.c | 1 +<br>
16 files changed, 1413 insertions(+), 278 deletions(-)<br>
<br>
-- <br>
2.20.1<br>
<br>
<br>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr">Marc-André Lureau<br></div></div>
</blockquote></div></div>