[Linux-cachefs] How capacious and well-indexed are ext4, xfs and btrfs directories?

[Andreas Dilger]

On May 25, 2021, at 4:31 PM, David Howells <dhowells at redhat.com> wrote:
> 
> Andreas Dilger <adilger at dilger.ca> wrote:
> 
>> As described elsewhere in the thread, allowing concurrent create and unlink
>> in a directory (rename probably not needed) would be invaluable for scaling
>> multi-threaded workloads.  Neil Brown posted a prototype patch to add this
>> to the VFS for NFS:
> 
> Actually, one thing I'm looking at is using vfs_tmpfile() to create a new file
> (or a replacement file when invalidation is required) and then using
> vfs_link() to attach directory entries in the background (possibly using
> vfs_link() with AT_LINK_REPLACE[1] instead of unlink+link).
> 
> Any thoughts on how that might scale?  vfs_tmpfile() doesn't appear to require
> the directory inode lock.  I presume the directory is required for security
> purposes in addition to being a way to specify the target filesystem.

I don't see how that would help much?  Yes, the tmpfile allocation would be
out-of-line vs. the directory lock, so this may reduce the lock hold time
by some fraction, but this would still need to hold the directory lock
when linking the tmpfile into the directory, in the same way that create
and unlink are serialized against other threads working in the same dir.

Having the directory locking scale with the size of the directory is what
will get orders of magnitude speedups for large concurrent workloads.
In ext4 this means write locking the directory leaf blocks independently,
with read locks for the interior index blocks unless new leaf blocks are
added (they are currently never removed).

It's the same situation as back with the BKL locking the entire kernel,
before we got fine-grained locking throughout the kernel.

> 
> David
> 
> [1] https://lore.kernel.org/linux-fsdevel/cover.1580251857.git.osandov@fb.com/
> 


Cheers, Andreas





-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 873 bytes
Desc: Message signed with OpenPGP
URL: <http://listman.redhat.com/archives/linux-cachefs/attachments/20210525/c6c98375/attachment.sig>