<html><body><div style="color:#000; background-color:#fff; font-family:arial, helvetica, sans-serif;font-size:12pt"><div><span><div>Joe,</div><div> Thanks for looking into the issue and running the tests and suggesting to use "direct" flag. I do see a difference with "direct" flag using dd. However the difference is significant when using bs=64M compared to bs=4k. </div><div><br></div><div><meta http-equiv="content-type" content="text/html; charset=utf-8"><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; "><span class="Apple-style-span" style="font-size: 16px; "><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; "><span class="Apple-style-span" style="font-size: 16px; "><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; ">dd-blocksize flags dm-thin LVMoutput </div><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; ">64M
none 179 MB/s 1GB/s </div><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; ">64M direct 2.4GB/s 3.6GB/s </div><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; ">4k none 179 MB/s 965MB/s </div><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; ">4k direct 193MB/s 1.7GB/s </div><div><br></div></span></div></span></div></div><div>In all the tests, dm-thin performance is below lvm. <span
class="Apple-style-span" style="font-size: 16px; ">I used 8M pages as we are not planning to use snapshots right now. </span></div><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; "><br></div><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; ">We are planning to use the dm-thin module in a high end flash device that supports upto 300k IOPS/second. Hence we are doing some performance tests. I wanted to try more parallel async IO instead of dd. A good program I found on web is <br></div><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; "><span class="Apple-style-span" style="color: rgb(0, 153, 51); font-family: arial, sans-serif; font-size: 14px; line-height: 15px; "><br></span></div><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; "><span class="Apple-style-span" style="color: rgb(0, 153, 51); font-family: arial, sans-serif; font-size: 14px;
line-height: 15px; ">fsbench.filesystems.org/bench/<b>aio</b>-<b>stress</b>.c</span><br></div><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; "><span class="Apple-style-span" style="color: rgb(0, 153, 51); font-family: arial, sans-serif; font-size: 14px; line-height: 15px; "><br></span></div><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; ">(if you are aware of other tools to test parallel async io using multiple threads, please let me know. I will give it a try.)<br></div><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; "><br></div><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; ">I compiled it with libaio and libpth. </div><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; "><br></div><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; ">Then I ran the test on LVM lun. Throughput is 9563.56MB/s. Fully
allocated TP LUN is 5399.81MB/s</div><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; "><br></div><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; ">I used "perf top" to see where most of the time is spent. Lot of time is spent in _raw_spin_lock. I poked the code little bit and I see that dm_thin_find_block returns EWOULDBLOCK and hence the bio gets deferred. It looks like spin locks are used when putting the bio on deferred list and getting it back from the deferred list. I see that EWOULDBLOCK is returned from dm_thin_find_block -> dm_btree_lookup->btree_lookup_raw->ro_step->bn_read_lock->dm_tm_read_lock->dm_bm_read_try_lock->bl_down_read_nonblock.</div><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; "><br></div><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; ">I tried to change the code little bit so
that dm_thin_find_block uses array as a cache and looks up btree only if not found in the array.</div><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; ">With that change, I see TP performance go way up.</div><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; "><br></div><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; ">Fully allocated TP LUN with array lookup is 9387.17MB/s. With array, no bios are being deferred. However when I used the array, I see kernel hang after several runs ( yet to debug the reason).</div><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; "><br></div><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; ">Code changes are initialize array in pool_ctr() in dm-thin.c</div><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; "><br></div><div>extern dm_block_t
blockmap[1000];<br></div><div><div>printk("blockmap array inited");</div><div> for(i=0;i<1000;i++)</div><div> {</div><div> blockmap[i]=0xffffffff;</div><div> }</div><div><br></div></div><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; "><br></div><div><div>dm_thin_find_block() in dm-thin-metadata.c</div><div><br></div><div>at the begining:</div><div><div> if(blockmap[block]==0xffffffff)</div><div> {</div><div> printk("found unassigned block %llu\n",block);</div><div> }</div><div> else</div><div> {</div><div> result->block =
blockmap[block];</div><div> result->shared = 0;</div><div> return 0;</div><div> }</div><div><br></div></div><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; ">at the end before returning:</div><div><div> if(r == 0)</div><div> {</div><div> blockmap[block] = result->block ;</div><div> }</div><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; "><br></div></div><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; "><br></div></div><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; ">Note that the array changes are only for my testing and not generic enough at all.
Command I used for getting the throughput numbers are</div><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; "><br></div><div>i=0; while [ $i -lt 100 ] ; do ./aio-stress.exe -O -o 1 -c 16 -t 16 -d 256 /dev/mapper/thin1 2>&1 | grep throughput | cut -f3 -d ' ' | cut -f2 -d '(' | cut -f1 -d ' ' ; let i=$i+1; done | awk 'BEGIN{i=0.0} {i+=$0} END{print i/100}'<br></div><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; "><br></div><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; ">Thanks,</div><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; ">Jagan.</div><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; "><br></div></span></div> <div style="font-size: 12pt; font-family: arial, helvetica, sans-serif; "> <div style="font-size: 12pt; font-family: 'times new roman', 'new york', times, serif; "> <div dir="ltr"> <font size="2"
face="Arial"> <hr size="1"> <b><span style="font-weight:bold;">From:</span></b> Joe Thornber <thornber@redhat.com><br> <b><span style="font-weight: bold;">To:</span></b> Jagan Reddy <gjmsreddy@yahoo.com>; device-mapper development <dm-devel@redhat.com> <br> <b><span style="font-weight: bold;">Sent:</span></b> Monday, January 16, 2012 4:42 AM<br> <b><span style="font-weight: bold;">Subject:</span></b> Re: [dm-devel] dm-thin vs lvm performance<br> </font> </div> <br>
Hi Jagan,<br><br>On Thu, Jan 12, 2012 at 04:55:16PM -0800, Jagan Reddy wrote:<br>> Hi,<br><br>> I recently started using dm-thin module and first of all, thank you<br>> for the good work. It works great, well documented and the comments<br>> in the code are very useful. I tried to run some performance tests<br>> using a high performance flash device and the performance of dm thin<br>> is about 30% of LVM performance on a full allocated thin provisioned<br>> volume. ( ie after all pages/blocks are allocated using<br>> dd). Performance test does only reads and no writes. <br><br>Thanks very much for taking the time to try thinp out properly.<br>People testing different scenarios is very useful to us.<br><br>You're probably aware that the thinp test suite is available here:<br><br> <a href="https://github.com/jthornber/thinp-test-suite"
target="_blank">https://github.com/jthornber/thinp-test-suite</a><br><br>I've added a little set of tests that recreate your scenario here:<br><br> <a href="https://github.com/jthornber/thinp-test-suite/blob/master/ramdisk_tests.rb" target="_blank">https://github.com/jthornber/thinp-test-suite/blob/master/ramdisk_tests.rb</a><br><br>I used a 2G ramdisk for these tests, and a variety of thinp block<br>sizes and 'dd' options. I'll just summarise the main results, also I<br>should point out that my testing was done on a VM hosted on a 4G<br>machine, so the machine was under a lot of memory pressure and there<br>was a lot of variance in the benchmarks.<br><br>writes across various volumes<br>-----------------------------<br><br>write1<br>------<br><br>Testing write performance.<br><br>dd if=/dev/zero of=/dev/mapper/<thin> oflags=direct bs=64M<br><br>zeroing new blocks turned on.<br><br>thinp block size = 64k<br><br>| Linear
| 2.2 G/s |<br>| Unprovisioned thin | 1.4 G/s |<br>| Provisioned thin | 1.9 |<br>| Snap totally shared | 1.5 G/s |<br>| Snap no sharing | 1.9 |<br><br>Pretty good. Not showing the drastic drop that you were seeing. The<br>small thinp block size means the snaps perform nicely (not many<br>copy-on-writes).<br><br>write2<br>------<br><br>As test1, but with 8M thinp block size as in your tests.<br><br>| Linear | 2.2 G/s |<br>| Unprovisioned thin | 1.5 G/s |<br>| Provisioned thin | 2.2 |<br>| Snap totally shared | 882 Ms |<br>| Snap no sharing | 2.2 | <br><br>Good results, breaking sharing performance is down because the large<br>block size mean there will be more actual copying incurred.<br><br>write3<br>------<br><br>As test2, but
no oflags=direct option to dd.<br><br>| Linear | 900 M/s |<br>| Unprovisioned thin | 579 M/s |<br>| Provisioned thin | 694 M/s |<br>| Snap totally shared | 510 M/s |<br>| Snap no sharing | 654 M/s | <br><br>Alarming. Results are similar for thinp block size of 64k.<br><br>read1<br>-----<br><br>Testing read performance.<br><br>dd if=/dev/mapper/<thin> of=/dev/null iflags=direct bs=64M<br><br>thinp block size = 64k<br><br>| Linear | 3.3 G/s |<br>| Provisioned thin | 2.7 G/s |<br>| Snap no sharing | 2.8 G/s | <br><br><br>read2<br>-----<br><br>read1 but with 8M thinp block size<br><br>| Linear | 3.3 G/s |<br>| Provisioned thin | 3.2 G/s |<br>| Snap no sharing | 3.3 G/s | <br><br>read3<br>-----<br><br>As read2, but
without the iflags=direct option to 'dd'.<br><br>| Linear | 1.0 G/s |<br>| Provisioned thin | 594 M/s |<br>| Snap no sharing | 605 M/s | <br><br><br><br>I think there are a couple of conclusions we can draw from this:<br><br>i) dd isn't great for benchmarking block devices<br>ii) if you are going to use it, then make sure you use O_DIRECT<br><br>Using an instrumented kernel, I've confirmed that these read tests<br>are taking the fast path through the code. The mapping tables are all<br>in memory. The bio_map function is always returning DM_MAPIO_REMAPPED.<br><br>Will you be using a filesystem on these block devices?<br><br>- Joe<br><br><br> </div> </div> </div></body></html>