<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">2016-06-21 16:59 GMT+08:00 Zdenek Kabelac <span dir="ltr"><<a href="mailto:zkabelac@redhat.com" target="_blank">zkabelac@redhat.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">Dne 21.6.2016 v 09:56 Dennis Yang napsal(a):<span class=""><br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
Hi,<br>
<br>
We have been dealing with a data corruption issue when we run out I/O<br>
test suite made by ourselves with multiple thin devices built on top of a<br>
thin-pool. In our test suites, we will create multiple thin devices and<br>
continually write to them, check the file checksum, and delete all files<br>
and issue DISCARD to reclaim space if no checksum error takes place.<br>
<br>
We found that there is one data access pattern could corrupt the data.<br>
Suppose that there are two thin devices A and B, and device A receives<br>
a DISCARD bio to discard a physical(pool) block 100. Device A will quiesce<br>
all previous I/O and held both virtual and physical data cell before it<br>
actually remove the corresponding data mapping. After the data mapping<br>
is removed, both data cell will be released and this DISCARD bio will<br>
be passed down to underlying devices. If device B tries to allocate<br>
a new block at the very same moment, it could reuse the block 100 which<br>
was just been discarded by device A (suppose metadata commit had<br>
been triggered, for a block cannot be reused in the same transaction).<br>
In this case, we will have a race between the WRITE bio coming from<br>
device B and the DISCARD bio coming from device A. Once the WRITE<br>
bio completes before the DISCARD bio, there would be checksum error<br>
for device B.<br>
<br>
So my question is, does dm-thin have any mechanism to eliminate the race when<br>
discarded block is reused right away by another device?<br>
<br>
Any help would be grateful.<br>
Thanks,<br>
</blockquote>
<br>
<br></span>
Please provide version of kernel and surrounding tools (OS release version)?<br>
also are you using 'lvm2' or you use directly 'dmsetup/ioctl' ?<br>
(in the later case we would need to see exact sequencing of operation).<br>
<br>
Also please provide reproducer script.<br>
<br>
<br>
Regards<br>
<br>
Zdenek<br>
<br>
--<br>
dm-devel mailing list<br>
<a href="mailto:dm-devel@redhat.com" target="_blank">dm-devel@redhat.com</a><br>
<a href="https://www.redhat.com/mailman/listinfo/dm-devel" rel="noreferrer" target="_blank">https://www.redhat.com/mailman/listinfo/dm-devel</a><br>
</blockquote></div><br><br>Hi Zdenek,</div><div class="gmail_extra"><br></div><div class="gmail_extra">We are using a customized dm-thin driver based on linux 3.19.8 running</div><div class="gmail_extra">on our QNAP NAS. Also, we create all our thin devices with "lvm2". I am</div><div class="gmail_extra">afraid that I cannot provide the reproducer script since we reproduce this by </div><div class="gmail_extra">running the I/O stress test suite on Windows to all thin devices exported to </div><div class="gmail_extra">them via samba and iSCSI. </div><div class="gmail_extra"><br></div><div class="gmail_extra">The following is the trace of thin-pool we dumped via blktrace. The data </div><div class="gmail_extra">corruption takes place from sector address 310150144 to 310150144 + 832.</div><div class="gmail_extra"> </div><div class="gmail_extra"><div class="gmail_extra">252,19 1 154916 184.875465510 29959 Q W 310150144 + 1024 [kworker/u8:0]</div><div class="gmail_extra">252,19 0 205964 185.496309521 0 C W 310150144 + 1024 [0]</div><div class="gmail_extra">^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^</div><div class="gmail_extra">At first, pool receives a 1024 sector WRITE bio which had allocated a pool block.</div><div class="gmail_extra"><br></div><div class="gmail_extra">252,19 3 353811 656.542481344 30280 Q D 310150144 + 1024 [kworker/u8:8]</div><div class="gmail_extra">^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^</div><div class="gmail_extra">Pool receives a 1024 sector (thin block size) DISCARD bio passed down by one of the thin device.</div><div class="gmail_extra"><br></div><div class="gmail_extra">252,19 1 495204 656.558652936 30280 Q W 310150144 + 832 [kworker/u8:8]</div><div class="gmail_extra">^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^</div><div class="gmail_extra">Another thin device passed down a 832 sector WRITE bio to the exact same place.</div><div class="gmail_extra"> </div><div class="gmail_extra">252,19 3 353820 656.564140283 0 C W 310150144 + 832 [0]</div><div class="gmail_extra">252,19 0 697455 656.770883592 0 C D 310150144 + 1024 [0]</div><div class="gmail_extra">^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^</div><div class="gmail_extra">Although the DISCARD bio was queued before the WRITE bio, their completion had </div><div class="gmail_extra">been reordered which could corrupt the data.</div><div class="gmail_extra"><br></div><div class="gmail_extra">252,19 1 515212 684.425478220 20751 A R 310150144 + 80 <- (252,22) 28932096</div><div class="gmail_extra">252,19 1 515213 684.425478325 20751 Q R 310150144 + 80 [smbd]</div><div class="gmail_extra">252,19 0 725274 684.425741079 23937 C R 310150144 + 80 [0]</div></div><div class="gmail_extra"><br></div><div class="gmail_extra">Hope this helps.</div><div class="gmail_extra">Thanks,</div><div class="gmail_extra"><br></div><div class="gmail_extra">Dennis</div><div class="gmail_extra"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><span lang="EN-US" style="font-size:10pt;font-family:Arial,sans-serif">Dennis Yang </span><span lang="EN-US" style="font-size:10pt;font-family:Arial,sans-serif"> </span><span lang="EN-US" style="font-size:10pt;font-family:Arial,sans-serif"><br>QNAP Systems, Inc.</span><div><span lang="EN-US" style="font-size:10pt;font-family:Arial,sans-serif">Skype: qnap.dennis.yang<br>Email:</span><span lang="EN-US" style="font-size:10pt;font-family:Arial,sans-serif"> <a href="mailto:dennisyang@qnap.com" target="_blank">dennisyang@qnap.com</a></span><div><span lang="EN-US" style="font-size:10pt;font-family:Arial,sans-serif">Tel:</span><span style="font-family:Arial,sans-serif;font-size:13.3333px"> (+886)-2-2393-5152 ext. </span><span style="font-family:Arial,sans-serif;font-size:13.3333px">15018</span><div><font face="Arial, sans-serif" size="2">Address: </font><span style="color:rgb(51,51,51);font-family:Verdana,Arial,Helvetica,sans-serif;font-size:12px;line-height:18.6667px">13F., No.56, Sec. 1, Xinsheng S. Rd., Zhongzheng Dist., Taipei City, Taiwan</span></div></div></div></div></div>
</div></div>