[Linux-cachefs] [PATCH v9 3/5] NFS: Convert buffered read paths to use netfs when fscache is enabled

[Daire Byrne]

On Mon, 14 Nov 2022 at 12:44, Benjamin Maynard <benmaynard at google.com> wrote:
>
> Thanks Dave for getting back to me so quickly.
>
> > Due to use of "drop_caches" this is almost certainly the known issue #1
> > I mentioned in the opening post of this series:
> > https://lore.kernel.org/all/20221103161637.1725471-1-dwysocha@redhat.com/
>
> Apologies, I completely missed the known issues in the original
> opening message of the series. Just to clarify, I was only ever
> dropping the caches on the "NFS Client" in the below relationship:
>
> Source NFS Server <-- Re-Export Server (with FS-Cache) <-- NFS Client.
>
> I never dropped the caches on the Re-Export Server (the server running
> FS-Cache) at any point.

So I have never actually done that particular test (I will need to
verify, but I think I would have noticed by now), but dropping caches
on the re-export server definitely caused repeat reads from "source"
NFS server and (re)writes to the fscache disk *without* David's
suggested patch. With the patch, you can drop caches on the re-export
server and get the repeat reads coming from the fscache disk as
expected.

You can certainly try that test too (just source NFS server ->
FS-cache client - read,drop cache,read).

I'm not sure about your particular test, but your re-export server
must have dropped the file from memory otherwise you would see the
repeat read just coming from page cache (with no fscache or disk cache
interaction required)? So I'll assume the file in question is too
large to fit into memory and effectively the cache has been dropped.
So I think the suggested patch on the re-export server will fix that
issue.

I should also add that I had this series working well (+suggested
patch) and the performance to/from disk cache is an order of magnitude
better than mainline (40MB/s vs 5000MB/s with NVMe), but it did expose
a race condition in the fscache use/unuse cookie code (David is
aware). In the NFS re-export case, we have lots of knfsd threads
thrashing around the netfs/fscache functions.

Daire