<div dir="ltr">forget the script, here it is ... </div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Feb 19, 2014 at 2:31 PM, George . <span dir="ltr"><<a href="mailto:george@ucdn.com" target="_blank">george@ucdn.com</a>></span> wrote:<br> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi,<div><br></div><div>I'm currently testing <span style="background-color:yellow"><span style="background-color:yellow"><span>dm</span></span></span>-cache on '<font color="#000000"><span style="background-color:yellow"><span style="background-color:yellow"><span>debian</span></span></span> 3.12-1-amd64 #1 <span style="background-color:yellow"><span style="background-color:yellow"><span>SMP</span></span></span> Debian 3.12.9-1' vanilla kernel and finally reproduced the issue with data corruption on cached device.</font></div> <div><font color="#000000"><br></font></div><div><font color="#000000">Using small and simple setup:</font></div><div><font color="#000000"><br></font></div><div><font color="#000000"><br></font></div><div><font color="#000000"><i><div> root@<span style="background-color:yellow"><span style="background-color:yellow"><span>debian</span></span></span>:~# <span style="background-color:yellow"><span style="background-color:yellow"><span>blockdev</span></span></span> --<span style="background-color:yellow"><span style="background-color:yellow"><span>getsz</span></span></span> /<span style="background-color:yellow"><span style="background-color:yellow"><span>dev</span></span></span>/sda5</div> <div>1951744</div><div>root@<span style="background-color:yellow"><span style="background-color:yellow"><span>debian</span></span></span>:~# <span style="background-color:yellow"><span style="background-color:yellow"><span>blockdev</span></span></span> --<span style="background-color:yellow"><span style="background-color:yellow"><span>getsz</span></span></span> /<span style="background-color:yellow"><span style="background-color:yellow"><span>dev</span></span></span>/sda6</div> <div>122880</div><div>root@<span style="background-color:yellow"><span style="background-color:yellow"><span>debian</span></span></span>:~# <span style="background-color:yellow"><span style="background-color:yellow"><span>blockdev</span></span></span> --<span style="background-color:yellow"><span style="background-color:yellow"><span>getsz</span></span></span> /<span style="background-color:yellow"><span style="background-color:yellow"><span>dev</span></span></span>/sda7</div> <div>28672</div><div><br></div></i></font></div><div><font color="#000000"><i>root@<span style="background-color:yellow"><span style="background-color:yellow"><span>debian</span></span></span>:~# <span style="background-color:yellow"><span style="background-color:yellow"><span>dmsetup</span></span></span> create cached --table '0 1951744 cache /<span style="background-color:yellow"><span style="background-color:yellow"><span>dev</span></span></span>/sda7 /<span style="background-color:yellow"><span style="background-color:yellow"><span>dev</span></span></span>/sda6 /<span style="background-color:yellow"><span style="background-color:yellow"><span>dev</span></span></span>/sda5 512 1 <span style="background-color:yellow"><span style="background-color:yellow"><span>writeback</span></span></span> default 0'</i><br> </font></div><div><font color="#000000"><br></font></div><div><font color="#000000">and a script that actually starts many (let's say 100) instances of <i><b><span style="background-color:yellow"><span style="background-color:yellow"><span>badblock</span></span></span></b></i> tool, writing, reading and comparing 0x55, </font><span style>0x</span><span style><span style="background-color:yellow"><span style="background-color:yellow"><span>aa</span></span></span>, </span><span style>0x</span><span style>ff, </span><span style>0x</span><span style>00 patterns, where each instance repeats that procedure over is own range of blocks. See my bash script (attached)</span></div> <div><font color="#000000"><br></font></div><div><font color="#000000">Here how I run it:</font></div><div><font color="#000000"><i>./<span style="background-color:yellow"><span>dm</span></span>-stress-test.sh -n 100 -r 10000 -d /<span style="background-color:yellow"><span>dev</span></span>/mapper/cached -t p</i><br> </font></div><div><br></div><div><div>/<span>dev</span>/mapper/cached, 512, 1951744</div><div>running parallel test</div><div>checking blocks 0 to 19516</div><div>checking blocks 19517 to 39033</div></div><div> <i style>.</i><br></div><div><font color="#000000"><i>.</i></font></div><div><font color="#000000"><i>.</i></font></div><div><font color="#000000"><div><i>checking blocks 1932183 to 1951699</i></div><div> <i>waiting for bad blocks</i></div><div><i>file has bad blocks, ./bad_block_43</i></div><div><i>839295</i></div><div><i>file has bad blocks, ./bad_block_43</i></div><div><i>839295</i></div><div><i>file has bad blocks, ./bad_block_43</i></div> <div><i>839295</i></div><div><i>^<span style="background-color:yellow"><span>CCTRL</span></span>-C exiting ...</i></div><div><br></div><div>here is the <span style="background-color:yellow"><span>dm</span></span>-cache status at the time of first corrupted block:</div> <div><br></div><div><div><i>root@<span style="background-color:yellow"><span>debian</span></span>:~# <span style="background-color:yellow"><span>dmsetup</span></span> status </i></div> <div><i>cached: 0 1951744 cache 7/3584 852 126848 1040 166813 0 85 85 71 0 2 migration_threshold 2048 4 random_threshold 4 sequential_threshold 51</i>2</div></div><div><br></div></font></div><div><font color="#000000"><br> </font></div><div><font color="#000000">I've also tried this test on <span style="background-color:yellow">custom</span> build <span style="background-color:yellow">kernel</span> 'Linux <span style="background-color:yellow"><span>george</span></span> 3.13.0+ #3 <span style="background-color:yellow"><span>SMP</span></span> Mon Feb 17 10:44:59 <span style="background-color:yellow"><span>EET</span></span> 2014 x86_64 x86_64 x86_64 GNU/Linux' and there is the same issue </font></div> <div><br></div><div><font color="#000000">If using ram device for cache and <span style="background-color:yellow"><span>sata</span></span> for origin the first corrupted blocks appears almost immediately .</font></div> <div><font color="#000000"><br></font></div><div><font color="#000000"><br></font></div><div><font color="#000000">We been testing <span style="background-color:yellow"><span style="background-color:yellow"><span>dm</span></span></span>-cache on larger setups (see <a href="https://www" target="_blank">https://www</a>.<span style="background-color:yellow"><span style="background-color:yellow"><span>redhat</span></span></span>.com/archives/<span style="background-color:yellow"><span style="background-color:yellow"><span>dm</span></span></span>-<span style="background-color:yellow"><span style="background-color:yellow"><span>devel</span></span></span>/2014-January/msg00135.html</font><span style>) but there we were able to corrupt cached device after several days under heavy traffic.</span></div> </div> </blockquote></div><br></div>