[Linux-cachefs] Adventures in NFS re-exporting

[Trond Myklebust]

On Thu, 2020-12-03 at 13:45 -0800, Frank Filz wrote:
> > On Thu, 2020-12-03 at 16:13 -0500, bfields at fieldses.org wrote:
> > > On Thu, Dec 03, 2020 at 08:27:39PM +0000, Trond Myklebust wrote:
> > > > On Thu, 2020-12-03 at 13:51 -0500, bfields wrote:
> > > > > I've been scratching my head over how to handle reboot of a
> > > > > re-
> > > > > exporting server.  I think one way to fix it might be just to
> > > > > allow the re- export server to pass along reclaims to the
> > > > > original
> > > > > server as it receives them from its own clients.  It might
> > > > > require
> > > > > some protocol tweaks, I'm not sure.  I'll try to get my
> > > > > thoughts
> > > > > in order and propose something.
> > > > > 
> > > > 
> > > > It's more complicated than that. If the re-exporting server
> > > > reboots,
> > > > but the original server does not, then unless that re-exporting
> > > > server persisted its lease and a full set of stateids
> > > > somewhere, it
> > > > will not be able to atomically reclaim delegation and lock
> > > > state on
> > > > the server on behalf of its clients.
> > > 
> > > By sending reclaims to the original server, I mean literally
> > > sending
> > > new open and lock requests with the RECLAIM bit set, which would
> > > get
> > > brand new stateids.
> > > 
> > > So, the original server would invalidate the existing client's
> > > previous clientid and stateids--just as it normally would on
> > > reboot--but it would optionally remember the underlying locks
> > > held by
> > > the client and allow compatible lock reclaims.
> > > 
> > > Rough attempt:
> > > 
> > > 
> > > https://wiki.linux-nfs.org/wiki/index.php/Reboot_recovery_for_re-expor
> > > t_servers
> > > 
> > > Think it would fly?
> > 
> > So this would be a variant of courtesy locks that can be reclaimed
> > by the client
> > using the reboot reclaim variant of OPEN/LOCK outside the grace
> > period? The
> > purpose being to allow reclaim without forcing the client to
> > persist the original
> > stateid?
> > 
> > Hmm... That's doable, but how about the following alternative: Add
> > a function
> > that allows the client to request the full list of stateids that
> > the server holds on
> > its behalf?
> > 
> > I've been wanting such a function for quite a while anyway in order
> > to allow the
> > client to detect state leaks (either due to soft timeouts, or due
> > to reordered
> > close/open operations).
> 
> Oh, that sounds interesting. So basically the re-export server would
> re-populate it's state from the original server rather than relying
> on it's clients doing reclaims? Hmm, but how does the re-export
> server rebuild its stateids? I guess it could make the clients
> repopulate them with the same "give me a dump of all my state", using
> the state details to match up with the old state and replacing
> stateids. Or did you have something different in mind?
> 

I was thinking that the re-export server could just use that list of
stateids to figure out which locks can be reclaimed atomically, and
which ones have been irredeemably lost. The assumption is that if you
have a lock stateid or a delegation, then that means the clients can
reclaim all the locks that were represented by that stateid.

I suppose the client would also need to know the lockowner for the
stateid, but presumably that information could also be returned by the
server?

-- 
Trond Myklebust
Linux NFS client maintainer, Hammerspace
trond.myklebust at hammerspace.com