[Virtio-fs] 'FORGET' ordering semantics (vs unlink & NFS)

[Vivek Goyal]

On Mon, Jan 04, 2021 at 02:04:39PM -0500, Vivek Goyal wrote:
> On Mon, Jan 04, 2021 at 06:56:55PM +0000, Dr. David Alan Gilbert wrote:
> > * Vivek Goyal (vgoyal at redhat.com) wrote:
> > > On Mon, Jan 04, 2021 at 04:00:13PM +0000, Dr. David Alan Gilbert wrote:
> > > > Hi,
> > > >   On virtio-fs we're hitting a problem with NFS, where
> > > > unlinking a file in a directory and then rmdir'ing that
> > > > directory fails complaining about the directory not being empty.
> > > > 
> > > > The problem here is that if a file has an open fd, NFS doesn't
> > > > actually delete the file on unlink, it just renames it to
> > > > a hidden file (e.g. .nfs*******).  That open file is there because
> > > > the 'FORGET' hasn't completed yet by the time the rmdir is issued.
> > > > 
> > > > Question:
> > > >   a) In the FUSE protocol, are requests assumed to complete in order;
> > > > i.e.  unlink, forget, rmdir   is it required that 'forget' completes
> > > > before the rmdir is processed?
> > > >      (In virtiofs we've been processing requests, in parallel, and
> > > > have sent forgets down a separate queue to keep them out of the way).
> > > > 
> > > >   b) 'forget' doesn't send a reply - so the kernel can't wait for the
> > > > client to have finished it;  do we need a synchronous forget here?
> > > 
> > > Even if we introduce a synchronous forget, will that really fix the
> > > issue. For example, this could also happen if file has been unlinked
> > > but it is still open and directory is being removed.
> > > 
> > > fd = open(foo/bar.txt)
> > > unlink foo/bar.txt
> > > rmdir foo
> > > close(fd).
> > > 
> > > In this case, final forget should go after fd has been closed. Its
> > > not a forget race. 
> > > 
> > > I wrote a test case for this and it works on regular file systems.
> > > 
> > > https://github.com/rhvgoyal/misc/blob/master/virtiofs-tests/rmdir.c
> > > 
> > > I suspect it will fail on nfs because I am assuming that temporary
> > > file will be there till final close(fd) happens. If that's the
> > > case this is a NFS specific issue because its behavior is different
> > > from other file systems.
> > 
> > That's true; but that's NFS just being NFS; in our case we're keeping
> > an fd open even though the guest has been smart enough not to; so we're
> > causing the NFS oddity when it wouldn't normally happen.
> 
> So what file descriptor is that? Is it because of O_PATH fd we have
> stashed away in lo_inode?
> 
> Will NFS keep .nfsXXXX file around even if O_PATH fd is open for
> unlinked file?

If this issue is due to O_PATH fd opened by virtiofsd, then I guess
we will have look into introducing synchronous and use that instead
when inode is being dropped (I guess in fuse_evict_inode()).

Vivek