<div dir="ltr"> Hi all, We are going to develop the v2 patch for virtio-fs crash reconnection. As suggested by Marc-André and Stefan, except for the inflight I/O tracking log area, all the other internal statuses of virtiofsd will be saved to some places other than QEMU. Specifically, the three lo_maps (ino_map, dirp_map, and fd_map) could be saved to several mmapped files, and the opened fds could be saved to systemd. I'd like to get some feedback on our further thoughts before we work on the revision. 1. What about by default save the opened fds as file handles to host kernel, instead of saving them to systemd. After some internal discussion, we think introducing systemd may introduce more uncertainness to the system, as we need to create one service for each daemon, and all the daemons may suffer the single-point failure of the systemd process. 2. Like the btree map implementation (<a href="http://multikey.rs">multikey.rs</a>) of virtiofsd-rs, what about splitting the flatten lo_map implementation, which supports to be persisted to files, from passhtrough_ll.c to a new separated source file. This way, maybe we can more easily wrap it with some Rust compatible interfaces, and enable crash recovery for virtiofsd-rs based on it. 3. What about dropping the dirp_map, and integrate the opened directory fds to fd_map. The virtiofsd-rs implementation only has two maps (inodes and handles). In the C version, dirp_map may also unnecessary. All the best, Jiachen</div> <div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Dec 16, 2020 at 12:21 AM Jiachen Zhang <<a href="mailto:zhangjiachen.jaycee@bytedance.com">zhangjiachen.jaycee@bytedance.com</a>> wrote: </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi, all We implement virtio-fs crash reconnection in this patchset. The crash reconnection of virtiofsd here is completely transparent to guest, no remount in guest is needed, even the inflight requests can be handled normally after reconnection. We are looking forward to any comments. Thanks, Jiachen OVERVIEW: To support virtio-fs crash reconnection, we need to support the recovery of 1) inflight FUSE request, and 2) virtiofsd internal status information. Fortunately, QEMU's vhost-user reconnection framework already supports inflight I/O tracking by using VHOST_USER_GET_INFLIGHT_FD and VHOST_USER_SET_INFLIGHT_FD (see 5ad204bf2 and 5f9ff1eff for details). As the FUSE requests are transferred by virtqueue I/O requests, by using the vhost-user inflight I/O tracking, we can recover the inflight FUSE requests. To support virtiofsd internal status recovery, we introduce 4 new vhost-user message types. As shown in the following diagram, two of them are used to persist shared lo_maps and opened fds to QEMU, the other two message types are used to restore the status when reconnecting. VHOST_USER_SLAVE_SHM VHOST_USER_SLAVE_FD +--------------+ Persist +--------------------+ | <---------------------+ | | QEMU | | Virtio-fs Daemon | | +---------------------> | +--------------+ Restore +--------------------+ VHOST_USER_SET_SHM VHOST_USER_SET_FD Although the 4 newly added message types are to support virtiofsd reconnection in this patchset, it might be potential in other situation. So we keep in mind to make them more general when add them to vhost related source files. VHOST_USER_SLAVE_SHM and VHOST_USER_SET_SHM can be used for memory sharing between a vhost-user daemon and QEMU, VHOST_USER_SLAVE_FD and VHOST_USER_SET_FD would be useful if we want to shared opened fds between QEMU process and vhost-user daemon process. USAGE and NOTES: - The commits are rebased to a recent QEMU master commit b4d939133dca0fa2b. - ",reconnect=1" should be added to the "-chardev socket" of vhost-user-fs-pci in the QEMU command line, for example: qemu-system-x86_64 ... \ -chardev socket,id=char0,path=/tmp/vhostqemu,reconnect=1 \ -device vhost-user-fs-pci,queue-size=1024,chardev=char0,tag=myfs \ ... - We add new options for virtiofsd to enable or disable crash reconnection. And some options are not supported by crash reconnection. So add following options to virtiofsd to enable reconnection: virtiofsd ... -o reconnect -o no_mount_ns -o no_flock -o no_posix_lock -o no_xattr ... - The reasons why virtiofsd-side locking, extended attributes, and mount namespace are not supported is explained in the commit message of the 6th patch (virtiofsd: Add two new options for crash reconnection). - The 9th patch is a work-around that will not affect the overall correctness. We remove the qsort related codes because we found that when resubmit_num is larger than 64, seccomp will kill the virtiofsd process. - Support for dax version virtiofsd is very possible and requires almost no additional change to this patchset. Jiachen Zhang (9): vhost-user-fs: Add support for reconnection of vhost-user-fs backend vhost: Add vhost-user message types for sending shared memory and file fds vhost-user-fs: Support virtiofsd crash reconnection libvhost-user: Add vhost-user message types for sending shared memory and file fds virtiofsd: Convert the struct lo_map array to a more flatten layout virtiofsd: Add two new options for crash reconnection virtiofsd: Persist/restore lo_map and opened fds to/from QEMU virtiofsd: Ensure crash consistency after reconnection virtiofsd: (work around) Comment qsort in inflight I/O tracking contrib/libvhost-user/libvhost-user.c | 106 +++- contrib/libvhost-user/libvhost-user.h | 70 +++ docs/interop/vhost-user.rst | 41 ++ hw/virtio/vhost-user-fs.c | 334 ++++++++++- hw/virtio/vhost-user.c | 123 ++++ hw/virtio/vhost.c | 42 ++ include/hw/virtio/vhost-backend.h | 6 + include/hw/virtio/vhost-user-fs.h | 16 +- include/hw/virtio/vhost.h | 42 ++ tools/virtiofsd/fuse_lowlevel.c | 24 +- tools/virtiofsd/fuse_virtio.c | 44 ++ tools/virtiofsd/fuse_virtio.h | 1 + tools/virtiofsd/helper.c | 9 + tools/virtiofsd/passthrough_helpers.h | 2 +- tools/virtiofsd/passthrough_ll.c | 830 ++++++++++++++++++-------- tools/virtiofsd/passthrough_seccomp.c | 1 + 16 files changed, 1413 insertions(+), 278 deletions(-) -- 2.20.1 </blockquote></div>