<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Oct 26, 2021 at 2:27 PM Vivek Goyal <<a href="mailto:vgoyal@redhat.com" target="_blank">vgoyal@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Tue, Oct 26, 2021 at 08:59:44PM +0300, Amir Goldstein wrote:<br>
> On Tue, Oct 26, 2021 at 7:18 PM Vivek Goyal <<a href="mailto:vgoyal@redhat.com" target="_blank">vgoyal@redhat.com</a>> wrote:<br>
> ><br>
> > On Tue, Oct 26, 2021 at 06:23:50PM +0300, Amir Goldstein wrote:<br>
> ><br>
> > [..]<br>
> > > > 3) The lifetime of the local watch in the guest kernel is very<br>
> > > > important. Specifically, there is a possibility that the guest does not<br>
> > > > receive remote events on time, if it removes its local watch on the<br>
> > > > target or deletes the inode (and thus the guest kernel removes the watch).<br>
> > > > In these cases the guest kernel removes the local watch before the<br>
> > > > remote events arrive from the host (virtiofsd) and as such the guest<br>
> > > > kernel drops all the remote events for the target inode (since the<br>
> > > > corresponding local watch does not exist anymore).<br>
> ><br>
> > So this is one of the issues which has been haunting us in virtiofs. If<br>
> > a file is removed, for local events, event is generated first and<br>
> > then watch is removed. But in case of remote filesystems, it is racy.<br>
> > It is possible that by the time event arrives, watch is already gone<br>
> > and application never sees the delete event.<br>
> ><br>
> > Not sure how to address this issue.<br>
> <br>
<br>
> Can you take me through the scenario step by step.<br>
> I am not sure I understand the exact sequence of the race.<br>
<br>
Ioannis, please correct me If I get something wrong. You know exact<br>
details much more than me.<br>
<br>
A. Say a guest process unlinks a file.<br>
B. Fuse sends an unlink request to server (virtiofsd)<br>
C. File is unlinked on host. Assume there are no other users so inode<br>
will be freed as well. And event will be generated on host and watch<br>
removed.<br>
D. Now Fuse server will send a unlink request reply. unlink notification<br>
might still be in kernel buffers or still be in virtiofsd or could<br>
be in virtiofs virtqueue.<br>
E. Fuse client will receive unlink reply and remove local watch.<br><br></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Fuse reply and notification event are now traveling in parallel on<br>
different virtqueues and there is no connection between these two. And<br>
it could very well happen that fuse reply comes first, gets processed<br>
first and local watch is removed. And notification is processed right<br>
after but by then local watch is gone and filesystem will be forced to<br>
drop event.<br>
<br>
As of now situation is more complicated in virtiofsd. We don't keep<br>
file handle open for file and keep an O_PATH fd open for each file.<br>
That means in step D above, inode on host is not freed yet and unlink<br>
event is not generated yet. When unlink reply reaches fuse client,<br>
it sends FORGET messages to server, and then server closes O_PATH fd<br>
and then host generates unlink events. By that time its too late,<br>
guest has already remove local watches (and triggered removal of<br>
remote watches too).<br>
<br>
This second problem probably can be solved by using file handles, but<br>
basic race will still continue to be there.<br>
<br>
> If it is local file removal that causes watch to be removed,<br>
> then don't drop local events and you are good to go.<br>
> Is it something else?<br>
<br>
- If remote events are enabled, then idea will be that user space gets<br>
and event when file is actually removed from server, right? Now it<br>
is possible that another VM has this file open and file has not been<br>
yet removed. So local event only tells you that file has been removed<br>
in guest VM (or locally) but does not tell anything about the state<br>
of file on server. (It has been unlinked on server but inode continues<br>
to be alive internall).<br>
<br>
- If user receives both local and remote delete event, it will be<br>
confusing. I guess if we want to see both the events, then there<br>
has to be some sort of info in event which classifies whether event<br>
is local or remote. And let application act accordingly.<br>
<br>
Thanks<br>
Vivek<br>
<br>
</blockquote></div><div><br>Hello Amir! </div><div><br></div><div>Sorry for taking part in the conversation a bit late. Vivek was on point with the</div><div>example he gave but the race is a bit more generic than only the DELETE event.</div><div><br></div><div>Let's say that a guest process monitors an inode for OPEN events:</div><div><br></div><div>1) The same guest process or another guest process opens the file (related to the</div><div>monitored inode), and then closes and immediately deletes the file/inode.</div><div>2) The FUSE server (virtiofsd) will mimic the operations of the guest process:</div><div> a) Will open the file on the host side and thus a remote OPEN event is going to </div><div> be generated on the host and sent to the guest.</div><div> b) Will unlink the remote inode and if no other host process uses the inode then the</div><div> inode will be freed and a DELETE event is going to be generated on the host and sent</div><div> to the guest (However, due to how virtiofsd works and Vivek mentioned, this step won't </div><div> happen immediately)</div><div><br></div><div>The problem here is that the OPEN event might still be travelling towards the guest in the</div><div>virtqueues and arrives after the guest has already deleted its local inode. </div><div>While the remote event (OPEN) received by the guest is valid, its fsnotify</div><div>subsystem will drop it since the local inode is not there.</div><div><br></div>-- <br><div dir="ltr"><div dir="ltr">Ioannis Angelakopoulos<div>Software Engineer Intern at Red Hat</div></div></div></div>