<html> <head> <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"> <style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style> </head> <body dir="ltr"> <div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> Hi Stefan,</div> <div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> <br> </div> <div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> Thanks for the explanation.</div> <div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> <br> </div> <div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> So reconnection for vhost-user is not a well defined behavior,</div> <div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> and QEMU is doing its best to retry when possible, depending</div> <div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> on each device.</div> <div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> <br> </div> <div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> The guest does not know about it, so it's never notified that</div> <div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> the device needs to be reset.</div> <div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> <br> </div> <div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> But what about the vhost-user backend initialization? Does</div> <div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> QEMU go again through initializing memory table, vrings, etc...</div> <div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> since it can't assume anything from the backend?</div> <div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> <br> </div> <div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> Thanks,</div> <div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> Sebastien<br> </div> <div> <div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0);"> <br> <hr tabindex="-1" style="display:inline-block; width:98%;"> <b>From:</b> Stefan Hajnoczi<br> <b>Sent:</b> Tuesday, May 11, 2021 2:45 PM<br> <b>To:</b> Boeuf, Sebastien<br> <b>Cc:</b> virtio-fs@redhat.com; qemu-devel@nongnu.org<br> <b>Subject:</b> vhost-user reconnection and crash recovery <div><br> </div> </div> <div class="BodyFragment"><font size="2"><span style="font-size:11pt;"> <div class="PlainText">Hi Sebastien,<br> On #virtio-fs IRC you asked:<br> <br> I have a vhost-user question regarding disconnection/reconnection. How<br> should this be handled? Let's say the vhost-user backend disconnects,<br> and reconnects later on, does QEMU reset the virtio device by notifying<br> the guest? Or does it simply reconnects to the backend without letting<br> the guest know about what happened?<br> <br> The vhost-user protocol does not have a generic reconnection solution.<br> Reconnection is handled on a case-by-case basis because device-specific<br> and implementation-specific state is involved.<br> <br> The vhost-user-fs-pci device in QEMU has not been tested with<br> reconnection as far as I know.<br> <br> The ideal reconnection behavior is to resume the device from its<br> previous state without disrupting the guest. Device state must survive<br> reconnection in order for this to work. Neither QEMU virtiofsd nor<br> virtiofsd-rs implement this today.<br> <br> virtiofs has a lot of state, making it particularly difficult to support<br> either DEVICE_NEEDS_RESET or transparent vhost-user reconnection. We<br> have discussed virtiofs crash recovery on the bi-weekly virtiofs call<br> (<a href="https://etherpad.opendev.org/p/virtiofs-external-meeting" target="_blank" rel="noopener noreferrer" data-auth="NotApplicable">https://etherpad.opendev.org/p/virtiofs-external-meeting</a>). If you want<br> to work on this then joining the call would be a good starting point to<br> coordinate with others.<br> <br> One approach for transparent crash recovery is for virtiofsd to keep its<br> state in tmpfs (e.g. inode/fd mappings) and open fds shared with a<br> clone(2) process via CLONE_FILES. This way the virtiofsd process can<br> terminate but its state persists in memory thanks to its clone process.<br> The clone can then be used to launch the new virtiofsd process from the<br> old state. This would allow the device to resume transparently with QEMU<br> only reconnecting the vhost-user UNIX domain socket. This is an idea<br> that we discussed in the bi-weekly virtiofs call.<br> <br> You mentioned device reset. VIRTIO 1.1 has the Device Status Field<br> DEVICE_NEEDS_RESET flat that the device can use to tell the driver that<br> a reset is necessary. This feature is present in the specification but<br> not implemented in the Linux guest drivers. Again the reason is that<br> handling it requires driver-specific logic for restoring state after<br> reset...otherwise the device reset would be visible to userspace.<br> <br> Stefan<br> </div> </span></font></div> </div> <p>---------------------------------------------------------------------<br> Intel Corporation SAS (French simplified joint stock company)<br> Registered headquarters: "Les Montalets"- 2, rue de Paris, <br> 92196 Meudon Cedex, France<br> Registration Number: 302 456 199 R.C.S. NANTERRE<br> Capital: 4,572,000 Euros</p> <p>This e-mail and any attachments may contain confidential material for<br> the sole use of the intended recipient(s). Any review or distribution<br> by others is strictly prohibited. If you are not the intended<br> recipient, please contact the sender and delete all copies.</p></body> </html>