[Linux-cachefs] Adventures in NFS re-exporting

[bfields at fieldses.org]

On Thu, Dec 03, 2020 at 09:57:41PM +0000, Trond Myklebust wrote:
> On Thu, 2020-12-03 at 13:45 -0800, Frank Filz wrote:
> > > On Thu, 2020-12-03 at 16:13 -0500, bfields at fieldses.org wrote:
> > > > On Thu, Dec 03, 2020 at 08:27:39PM +0000, Trond Myklebust wrote:
> > > > > On Thu, 2020-12-03 at 13:51 -0500, bfields wrote:
> > > > > > I've been scratching my head over how to handle reboot of a
> > > > > > re-
> > > > > > exporting server.  I think one way to fix it might be just to
> > > > > > allow the re- export server to pass along reclaims to the
> > > > > > original
> > > > > > server as it receives them from its own clients.  It might
> > > > > > require
> > > > > > some protocol tweaks, I'm not sure.  I'll try to get my
> > > > > > thoughts
> > > > > > in order and propose something.
> > > > > > 
> > > > > 
> > > > > It's more complicated than that. If the re-exporting server
> > > > > reboots,
> > > > > but the original server does not, then unless that re-exporting
> > > > > server persisted its lease and a full set of stateids
> > > > > somewhere, it
> > > > > will not be able to atomically reclaim delegation and lock
> > > > > state on
> > > > > the server on behalf of its clients.
> > > > 
> > > > By sending reclaims to the original server, I mean literally
> > > > sending
> > > > new open and lock requests with the RECLAIM bit set, which would
> > > > get
> > > > brand new stateids.
> > > > 
> > > > So, the original server would invalidate the existing client's
> > > > previous clientid and stateids--just as it normally would on
> > > > reboot--but it would optionally remember the underlying locks
> > > > held by
> > > > the client and allow compatible lock reclaims.
> > > > 
> > > > Rough attempt:
> > > > 
> > > > 
> > > > https://wiki.linux-nfs.org/wiki/index.php/Reboot_recovery_for_re-expor
> > > > t_servers
> > > > 
> > > > Think it would fly?
> > > 
> > > So this would be a variant of courtesy locks that can be reclaimed
> > > by the client
> > > using the reboot reclaim variant of OPEN/LOCK outside the grace
> > > period? The
> > > purpose being to allow reclaim without forcing the client to
> > > persist the original
> > > stateid?
> > > 
> > > Hmm... That's doable, but how about the following alternative: Add
> > > a function
> > > that allows the client to request the full list of stateids that
> > > the server holds on
> > > its behalf?
> > > 
> > > I've been wanting such a function for quite a while anyway in order
> > > to allow the
> > > client to detect state leaks (either due to soft timeouts, or due
> > > to reordered
> > > close/open operations).
> > 
> > Oh, that sounds interesting. So basically the re-export server would
> > re-populate it's state from the original server rather than relying
> > on it's clients doing reclaims? Hmm, but how does the re-export
> > server rebuild its stateids? I guess it could make the clients
> > repopulate them with the same "give me a dump of all my state", using
> > the state details to match up with the old state and replacing
> > stateids. Or did you have something different in mind?
> > 
> 
> I was thinking that the re-export server could just use that list of
> stateids to figure out which locks can be reclaimed atomically, and
> which ones have been irredeemably lost. The assumption is that if you
> have a lock stateid or a delegation, then that means the clients can
> reclaim all the locks that were represented by that stateid.

I'm confused about how the re-export server uses that list.  Are you
assuming it persisted its own list across its own crash/reboot?  I guess
that's what I was trying to avoid having to do.

--b.