[dm-devel] Optane nvdimm performance

[Bruggeman, Otto (external - Partner)]

My apologies, I meant to forward this mail and managed to press the wrong button...

-----Original Message-----
From: Bruggeman, Otto (external - Partner) <otto.bruggeman at sap.com> 
Sent: Monday, March 30, 2020 2:38 PM
To: Mikulas Patocka <mpatocka at redhat.com>; Dan Williams <dan.j.williams at intel.com>; Vishal Verma <vishal.l.verma at intel.com>; Dave Jiang <dave.jiang at intel.com>; Ira Weiny <ira.weiny at intel.com>; Mike Snitzer <msnitzer at redhat.com>
Cc: linux-nvdimm at lists.01.org; dm-devel at redhat.com
Subject: [CAUTION] RE: Optane nvdimm performance

FYI Mal sehen was da an antworten kommen...

-----Original Message-----
From: Mikulas Patocka <mpatocka at redhat.com> 
Sent: Sunday, March 29, 2020 10:26 PM
To: Dan Williams <dan.j.williams at intel.com>; Vishal Verma <vishal.l.verma at intel.com>; Dave Jiang <dave.jiang at intel.com>; Ira Weiny <ira.weiny at intel.com>; Mike Snitzer <msnitzer at redhat.com>
Cc: linux-nvdimm at lists.01.org; dm-devel at redhat.com
Subject: Optane nvdimm performance

Hi

I performed some microbenchmarks on a system with real Optane-based nvdimm 
and I found out that the fastest method how to write to persistent memory 
is to fill a cacheline with 8 8-byte writes and then issue clwb or 
clflushopt on the cacheline. With this method, we can achieve 1.6 GB/s 
throughput for linear writes. On the other hand, non-temporal writes 
achieve only 1.3 GB/s.

The results are here:
http://people.redhat.com/~mpatocka/testcases/pmem/microbenchmarks/pmem.txt

The benchmarks here:
http://people.redhat.com/~mpatocka/testcases/pmem/microbenchmarks/

The winning benchmark is this:
http://people.redhat.com/~mpatocka/testcases/pmem/microbenchmarks/thrp-write-8-clwb.c


However, the kernel is not using this fastest method, it is using 
non-temporal stores instead.


I took the novafs filesystem (see git clone 
https://github.com/NVSL/linux-nova), it uses 
__copy_from_user_inatomic_nocache, which calls __copy_user_nocache which 
performs non-temporal stores. I hacked __copy_user_nocache to use clwb 
instead of non-temporal stores and it improved filesystem performance 
significantly.

This is the patch
http://people.redhat.com/~mpatocka/testcases/pmem/benchmarks/copy-nocache.patch 
(for the kernel 5.1 because novafs needs this version) and these are 
benchmark results:
http://people.redhat.com/~mpatocka/testcases/pmem/benchmarks/fs-bench.txt

- you can see that "test2" has twice the write throughput of "test1"


I took the dm-writecache driver, it uses memcpy_flushcache to write data 
to persistent memory. I hacked memcpy_flushcache to use clwb instead of 
non-temporal stores.

The result is - for 512-byte writes, non-temporal stores perform better 
than cache flushing. For 1024-byte and larger writes, cache flushing 
performs better than non-temporal stores. (I also tried to use cached 
writes + clwb for dm-writecache metadata updates, but it had bad 
performance)


Do you have some explanation why nontemporal stores are better for 
512-byte copies and worse for 1024-byte copies? (like filling up some 
buffers inside the CPU)?

In the next email, I'm sending a patch that makes memcpy_flushcache use 
clflushopt for transfers larger than 768 bytes.


Mikulas
_______________________________________________
Linux-nvdimm mailing list -- linux-nvdimm at lists.01.org
To unsubscribe send an email to linux-nvdimm-leave at lists.01.org
_______________________________________________
Linux-nvdimm mailing list -- linux-nvdimm at lists.01.org
To unsubscribe send an email to linux-nvdimm-leave at lists.01.org