[dm-devel] Snapshot target and DAX-capable devices

[Dave Chinner]

On Thu, Aug 30, 2018 at 02:49:07PM -0400, Mike Snitzer wrote:
> On Thu, Aug 30 2018 at  5:30am -0400,  Jan Kara <jack at suse.cz> wrote:
> > Well, changing device from DAX-capable to DAX-incapable is problematic for
> > filesystem on top of it as well. Filesystems simply don't expect this
> > feature of a device can change so they would fail in unexpected ways. Also
> > PFNs from the pmem (DAX-capable) device that are already mapped to user page
> > tables won't magically become unmapped so those processes will still have
> > DAX access to those areas of the device.
....
> As you point out, how are the upper layers (e.g. filesystems) supposed
> to reliably cope with this runtime switch to from DAX to non-DAX access?

They can't right now. There's unsolved races between page faults,
invalidations and changing the file operations to/from DAX
dynamically. This is the entire problem facing the dynamic per-inode
DAX on/off flag - if it happens globally to the filesystem without
warning, then the filesystem is screwed. 

To support the block device changing between DAX and non-DAX
dynamically, then the filesystem needs to first invalidate the entire
filesystem cache, eject all cached inodes from memory, any cached
metadata that is using DAX, etc to clear out all the DAX mappings
it have. And it has to do it without racing with new page faults or
IO that might map new DAX pages. And I'm ignoring the fact that we
can't eject referenced inodes (i.e. open files) from the inode cache
and so we currently cannot safely change the DAX on such files.
That's a blocker right now.

Once we can safely change the DAX state of open files, we've got to
co=ordinate the block device state change with the filesystem - the
filesystem wide invalidation has to be done before the block device can
start the change of state, and the filesystem must remain completely
stopped until the block device has completed it's change of state.

So AFAICT this ends up being "stop the world instantly, eject the
world from memory, rebuild the world from scratch, start the world
again".  Freezing the filesystem doesn't stop the world - we can
still do read IO and page faults, so that doesn't prevent pagefault
races with the invalidation leaving DAX references in the page
cache. Hence we currently have no valid "stop the world" mechanism
in the kernel other than unmount, which we can't do while there are
open files.

What about MAP_SYNC applications? If we turn off DAX with those
applications still running, we silently break them and users won't
know until the system loses power and they see data corruption after
the system comes back. However, applications SEGVing unpredictably
becuse of "transparent" storage state changes is almost as
unfriendly.

Dynamically changing block device DAX support seems like a
non-starter to me. At least, it's a non starter until we add a lot
more infrastructure, solve a bunch of really hard problems and
define how active userspace controlled DAX-only features behave when
DAX is no longer available...

Cheers,

Dave.
-- 
Dave Chinner
david at fromorbit.com