<div dir="ltr"><div><div><div>Okay, update on the problem.<br><br></div>Tested with <br>Linux v-5-231-d1862-150 3.13.0-rc8 #1 SMP Fri Jan 17 17:55:34 GMT 2014 x86_64 Intel(R) Xeon(R) CPU E5620 @ 2.40GHz GenuineIntel GNU/Linux<br>
<br><br></div>dm-cache breaks when cache gets full, again<br><br></div><div>dmesg:<br><br>Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589304] XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br>Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589310] XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br>
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589312] XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br>Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589315] XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br>
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589317] XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br>Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589319] XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br>
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589322] XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br>Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589324] XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br>
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589326] XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br>Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589328] XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br>
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589331] XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br>Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589333] XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br>
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589335] XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br>Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589337] XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br>
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589339] XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br>Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589342] XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br>
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589344] XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br>Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589346] XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br>
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589349] XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br>Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589351] XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br>
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589353] XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br>Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589355] XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br>
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589357] XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br>Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589360] XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br>
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589362] XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br>Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589364] XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br>
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589366] XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br>Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589368] XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br>
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589371] XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br>Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589373] XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br>
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589375] XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br>Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589377] XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br>
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589448] XFS (dm-0): metadata I/O error: block 0x2aeb4ee40 ("xfs_trans_read_buf_map") error 117 numblks 16<br>Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.589455] XFS (dm-0): xfs_imap_to_bp: xfs_trans_read_buf() returned error 117.<br>
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590202] XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br>Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590206] XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br>
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590209] XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br>Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590212] XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br>
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590214] XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br>Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590216] XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br>
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590218] XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br>Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590221] XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br>
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590223] XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br>Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590225] XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br>
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590227] XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br>Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590229] XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br>
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590232] XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br>Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590234] XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br>
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590236] XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br>Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590238] XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br>
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590241] XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br>Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590243] XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br>
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590245] XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br>Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590248] XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br>
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590250] XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br>Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590252] XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br>
Jan 22 00:34:55 v-5-231-d1862-150 kernel: [362737.590254] XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br><br></div><div>and so on... it just repeats the same<br><br> blockdev --report<br>RO RA SSZ BSZ StartSec Size Device<br>
rw 256 512 4096 0 26843414528 /dev/sda<br>rw 256 512 4096 2048 4094689280 /dev/sda1<br>rw 256 512 4096 7999488 22479372288 /dev/sda2<br>rw 2048 512 4096 0 20973392756736 /dev/sdb<br>
rw 2048 512 4096 2048 20973390659584 /dev/sdb1<br>rw 2048 512 4096 0 2397799710720 /dev/sdc<br>rw 2048 512 4096 2048 1023410176 /dev/sdc1<br>rw 2048 512 4096 2000896 2396774203392 /dev/sdc2<br>
rw 2048 512 512 0 20973390659584 /dev/dm-0<br><br><br> dmsetup table<br>storage: 0 40963653632 cache 8:33 8:34 8:17 8192 1 writeback default 0<br><br>dmsetup status<br>storage: 0 40963653632 cache 1152/249856 86805905 9729090 1164931 47563307 0 525209 525209 0 1 writeback 2 migration_threshold 10000000 4 random_threshold 4 sequential_threshold 10000000<br>
<br><br></div><div>Cache size ~2.2T, origin ~20TB<br><br></div><div>If you need more specific info, please tell me what.<br><br></div><div>reagrds.<br><br></div><div><div><br><br></div></div></div><div class="gmail_extra">
<br><br><div class="gmail_quote">On Wed, Nov 13, 2013 at 12:24 PM, Vladimir Smolensky <span dir="ltr"><<a href="mailto:arizal@gmail.com" target="_blank">arizal@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div><div><div><div>Hello,<br></div>I've been testing dm-cache for use with static web content.<br></div>It appears that, when using big cache ~3TB(+20TB origin dev) dm-cache device corrupts the filesystem after block eviction starts to happen.<br>
</div>If I set smaller cache size - single 480GB ssd(again with 20TB origin dev), the dm-cache dev works just fine.<br><br><br></div>dmesg error:<br><br>XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br>XFS (dm-0): metadata I/O error: block 0x3d4f534e0 ("xfs_trans_read_buf_map") error 117 numblks 16<br>
XFS (dm-0): xfs_imap_to_bp: xfs_trans_read_buf() returned error 117.<br>XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br>XFS (dm-0): Corruption detected. Unmount and run xfs_repair<br><div><br><br></div><div>
ext4 also got broken.<br></div><div><br><br></div><div>Tested with 3.10.11-gentoo, fedora 19- kernel 3.11.something, xfs, ext4, writeback and writethrough modes.<br><br><br></div><div><br>regards,<br></div><div>Vladimir Smolensky<br>
</div></div>
</blockquote></div><br></div>