[Virtio-fs] [PATCH v6 2/7] fuse: make DAX mount option a tri-state

[Ira Weiny]

On Mon, Oct 25, 2021 at 03:33:31PM -0400, Vivek Goyal wrote:

[snip]

> > > > > > 
> > > > > 
> > > > > I can only find the following discussions about the earliest record on
> > > > > this tri-state mount option:
> > > > > 
> > > > > https://lore.kernel.org/lkml/20200316095509.GA13788@lst.de/
> > > > > https://lore.kernel.org/lkml/20200401040021.GC56958@magnolia/
> > > > > 
> > > > > 
> > > > > Hi, Ira Weiny,
> > > > > 
> > > > > Do you have any thought on this, i.e. why the default behavior has
> > > > > changed after introduction of per inode dax?
> > > > 
> > > > While this is 'technically' different behavior the end user does not see any
> > > > difference in behavior if they continue without software changes.  Specifically
> > > > specifying nothing continues to operate with all the files on the FS to be
> > > > '_not_ DAX'.  While specifying '-o dax' forces DAX on all files.
> > > > 
> > > > This expands the default behavior in a backwards compatible manner.
> > > 
> > > This is backward compatible in a sense that if somebody upgrades to new
> > > kernel, things will still be same. 
> > > 
> > > I think little problematic change is that say I bring in persistent
> > > memory from another system (which has FS_XFLAGS_DAX set on some inodes)
> > > and then mount it without andy of the dax mount options, then per
> > > inode dax will be enabled unexpectedly if I boot with newer kernels
> > > but it will be disable if I mount with older kernels. Do I understand it
> > > right.
> > 
> > Indeed that will happen.  However, wouldn't the users (software) of those files
> > have knowledge that those files were DAX and want to continue with them in that
> > mode?
> 
> I am not sure. Say before per-inode dax feature, I had written a script
> which walks though all the mount points and figure out if dax is enabled
> or not. I could simply look at mount options and tell if dax could be
> enabled or not.
> 
> But now same script will give false results as per inode dax could
> still be enabled.

The mount option is being deprecated.  So it is best to start to phase out
scripts like that.

> 
> > 
> > > 
> > > > The user
> > > > can now enable DAX on some files.  But this is an opt-in on the part of the
> > > > user of the FS and again does not change with existing software/scripts/etc.
> > > 
> > > Don't understand this "opt-in" bit. If user mounts an fs without
> > > specifying any of the dax options, then per inode dax will still be
> > > enabled if inode has the correct flag set.
> > 
> > But only users who actually set that flag 'opt-in'.
> > 
> > > So is setting of flag being
> > > considered as opt-in (insted of mount option).
> > 
> > Yes.
> > 
> > > 
> > > If setting of flag is being considered as opt-in, that probably will not
> > > work very well with virtiofs. Because server can enforce a different
> > > policy for enabling per file dax (instead of FS_XFLAG_DAX).
> > 
> > I'm not sure I understand how this happens?  I think the server probably has to
> > enable per INODE by default to allow the client to do what the end users wants.
> > 
> 
> Server can have either per inode disabled or enabled. If enabled, it could
> determine DAX status of file based on FS_XFLAG_DAX or based on something
> else depending on server policy. Users want to be able to determine
> DAX status of file based on say file size.

'file size'?  I'm not sure how that would work.  Did you mean something else?

> 
> > I agree that if the end user is expecting DAX and the server disables it then
> > that is a problem but couldn't that happen before?
> 
> If end user expects to enable DAX and sever can't enable it, then mount
> fails. So currently if you mount "-o dax" and server does not support
> DAX, mount will fail.

The same could happen on a server where the underlying device does not support
DAX.  What if the server was mounted without '-o dax'?  Wouldn't a client mount
with '-o dax' fail now?  So why can't the same be true with the new set of
options?

> 
> I think same should happen when per inode DAX is introduced for virtiofs.
> If sever does not support per inode dax and user mounts with "-o
> dax=inode", then mount should fail.

I think that is reasonable.  The client can't mount with something the server
can't support.

> 
> In fact, this is another reason that probably "dax=inode" should not be
> default. Say client is new and server is old and does not support
> per inode dax, then client might start failing mount after client
> upgrade, and that's not good.

Shouldn't the client fall back to whatever the server supports?  It is the same
as the client wanting DAX now without server and/or device support.  It just
can't get it.  Right?

> 
> More I think about it, more it feels like that "dax=never" should be
> the default if user has not specified any of the dax options. This
> probably will introduce least amount of surprise. Atleast for virtiofs.
> IMHO, it probably would have made sense even for ext4/xfs but that
> ship has already sailed.

I disagree because dax=never is backwards from what we really want for the
future.  'dax=inode' is the most flexible setting.  In fact that setting is
best for the server by default which allows more control to be in the clients
hands.  Would you agree?

> 
> > Maybe I'm getting confused
> > because I'm not familiar enough with virtiofs.
> > 
> > > 
> > > And given there are two entities here (client and server), I think it
> > > will be good if if we give client a chance as well to decide whether
> > > it wants to enable per file dax or not. I know it can alwasy do 
> > > "dax=never" but it can still be broken if client software remains
> > > same but host/server software is upgraded or commnad line changed.
> > 
> > But the files are 'owned' by a single user or group of users who must have
> > placed the file in DAX mode at some point right?
> 
> Yes, either users/groups/admin might have set FS_XFLAG_DAX on inodes. But
> now there is another controller (virtiofs server) which determines whether
> that flag takes affect or not (based on server settings).

I think this is just like the file being on a device which does not support
DAX.  The file inode flag can be set but the file will not be in DAX mode on a
non-dax device.  So in this case the server is a non-dax device.

Ira

> 
> We did not have this server scenario in case of local filesystems.
> 
> Thanks
> Vivek
> >
> > > 
> > > So for virtiofs, I think better behavior is to continue to not enable
> > > any dax until and unless user opts-in using "-o dax=foo" options.
> > 
> > I'm not sure, maybe.
> > 
> > Ira
> > 
>