<div dir="ltr">Hi all,<div><br></div><div>The dmcache setup on one production server crashed this afternoon. The error in syslog is below.</div><div><br></div><div>After a reboot, logical volumes are still here, but inactive.</div><br># lvs -a<br> LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert<br> cache storage Cwi---C--- 834.41g <br> [cache_cdata] storage Cwi------- 834.41g <br> [cache_cmeta] storage ewi------- 16.00g <br> [lvol0_pmspare] storage ewi------- 16.00g <br> tank storage Cwi---C--- 5.00t cache [tank_corig] <br> [tank_corig] storage owi---C--- 5.00t <div> <br></div><div><div>I'm not able to activate them.</div><div><br></div># vgchange -ay<br> Check of pool storage/cache failed (status:1). Manual repair required!<br> 0 logical volume(s) in volume group "storage" now active</div><div><br></div><div>Any idea to bring back the volume online, or at least to recover the data ?</div><div><br></div><div>Kernel : 4.3.0-0.bpo.1-amd64 #1 SMP Debian 4.3.5-1~bpo8+1 (2016-02-23) x86_64 GNU/Linux</div><div>Distro : debian jessie</div><div><br></div>
<div>Any help will be greatly appreciated.</div><div><br></div><div>Regards,</div><div><br></div><div>Bertrand<br><div><br></div><div>Error log : </div><div><div>Jun 6 11:58:06 xxxx kernel: [3015831.571379] device-mapper: array: array_block_check failed: blocknr 1082331758718 != wanted 16969</div><div>Jun 6 11:58:06 xxxx kernel: [3015831.571453] device-mapper: block manager: array validator check failed for block 16969</div><div>Jun 6 11:58:06 xxxx kernel: [3015831.571515] device-mapper: cache: 252:3: demotion failed; couldn't update on disk metadata</div><div>Jun 6 11:58:06 xxxx kernel: [3015831.571578] device-mapper: cache: 252:3: metadata operation 'dm_cache_remove_mapping' failed: error = -15</div><div>Jun 6 11:58:06 xxxx kernel: [3015831.571664] device-mapper: cache: 252:3: aborting current metadata transaction</div><div>Jun 6 11:58:06 xxxx kernel: [3015831.572836] device-mapper: cache: 252:3: switching cache to read-only mode</div><div>Jun 6 11:58:06 xxxx kernel: [3015831.572840] device-mapper: cache: 252:3: demotion failed; couldn't update on disk metadata</div><div>Jun 6 11:58:06 xxxx kernel: [3015831.572901] device-mapper: cache: 252:3: metadata operation 'dm_cache_remove_mapping' failed: error = -22</div><div>Jun 6 11:58:06 xxxx kernel: [3015831.572973] device-mapper: cache: 252:3: demotion failed; couldn't update on disk metadata</div><div>Jun 6 11:58:06 xxxx kernel: [3015831.573034] device-mapper: cache: 252:3: metadata operation 'dm_cache_remove_mapping' failed: error = -22</div><div>Jun 6 11:58:06 xxxx kernel: [3015831.573103] device-mapper: cache: 252:3: demotion failed; couldn't copy block</div><div>Jun 6 11:58:11 xxxx kernel: [3015836.425768] Aborting journal on device dm-3-8.</div><div>Jun 6 11:58:11 xxxx kernel: [3015836.425853] Buffer I/O error on dev dm-3, logical block 671121408, lost sync page write</div><div>Jun 6 11:58:11 xxxx kernel: [3015836.425939] JBD2: Error -5 detected when updating journal superblock for dm-3-8.</div><div>Jun 6 11:58:11 xxxx kernel: [3015836.444140] Buffer I/O error on dev dm-3, logical block 0, lost sync page write</div><div>Jun 6 11:58:11 xxxx kernel: [3015836.444234] EXT4-fs error (device dm-3): ext4_journal_check_start:56: Detected aborted journal</div><div>Jun 6 11:58:11 xxxx kernel: [3015836.444311] EXT4-fs (dm-3): Remounting filesystem read-only</div><div>Jun 6 11:58:11 xxxx kernel: [3015836.444352] EXT4-fs (dm-3): previous I/O error to superblock detected</div><div>Jun 6 11:58:11 xxxx kernel: [3015836.444428] Buffer I/O error on dev dm-3, logical block 0, lost sync page write</div><div>Jun 6 11:58:15 xxxx kernel: [3015840.507213] migration_success_pre_commit: 12194 callbacks suppressed</div><div>Jun 6 11:58:15 xxxx kernel: [3015840.507217] device-mapper: cache: 252:3: promotion failed; couldn't update on disk metadata</div><div>Jun 6 11:58:15 xxxx kernel: [3015840.507291] device-mapper: cache: 252:3: metadata operation 'dm_cache_insert_mapping' failed: error = -22</div><div>Jun 6 11:58:19 xxxx kernel: [3015844.720334] device-mapper: cache: 252:3: promotion failed; couldn't update on disk metadata</div><div>Jun 6 11:58:19 xxxx kernel: [3015844.720406] device-mapper: cache: 252:3: metadata operation 'dm_cache_insert_mapping' failed: error = -22</div><div>Jun 6 11:59:15 xxxx kernel: [3015901.091051] device-mapper: cache: 252:3: demotion failed; couldn't update on disk metadata</div><div>Jun 6 11:59:15 xxxx kernel: [3015901.091124] device-mapper: cache: 252:3: metadata operation 'dm_cache_remove_mapping' failed: error = -22</div><div>Jun 6 11:59:15 xxxx kernel: [3015901.091207] device-mapper: cache: 252:3: demotion failed; couldn't update on disk metadata</div><div>Jun 6 11:59:15 xxxx kernel: [3015901.091274] device-mapper: cache: 252:3: metadata operation 'dm_cache_remove_mapping' failed: error = -22</div><div>Jun 6 11:59:15 xxxx kernel: [3015901.091482] device-mapper: cache: 252:3: demotion failed; couldn't update on disk metadata</div><div>Jun 6 11:59:15 xxxx kernel: [3015901.091551] device-mapper: cache: 252:3: metadata operation 'dm_cache_remove_mapping' failed: error = -22</div><div>Jun 6 11:59:15 xxxx kernel: [3015901.091622] device-mapper: cache: 252:3: demotion failed; couldn't update on disk metadata</div><div>Jun 6 11:59:15 xxxx kernel: [3015901.091689] device-mapper: cache: 252:3: metadata operation 'dm_cache_remove_mapping' failed: error = -22</div><div>Jun 6 11:59:15 xxxx kernel: [3015901.092341] device-mapper: cache: 252:3: demotion failed; couldn't update on disk metadata</div><div>Jun 6 11:59:15 xxxx kernel: [3015901.092409] device-mapper: cache: 252:3: metadata operation 'dm_cache_remove_mapping' failed: error = -22</div><div>Jun 6 11:59:30 xxxx kernel: [3015915.162952] migration_success_pre_commit: 1194 callbacks suppressed</div><div>Jun 6 11:59:30 xxxx kernel: [3015915.162955] device-mapper: cache: 252:3: promotion failed; couldn't update on disk metadata</div><div>Jun 6 11:59:30 xxxx kernel: [3015915.163034] device-mapper: cache: 252:3: metadata operation 'dm_cache_insert_mapping' failed: error = -22</div><div>Jun 6 11:59:38 xxxx kernel: [3015923.147099] device-mapper: cache: 252:3: promotion failed; couldn't update on disk metadata</div><div>Jun 6 11:59:38 xxxx kernel: [3015923.147183] device-mapper: cache: 252:3: metadata operation 'dm_cache_insert_mapping' failed: error = -22</div><div>Jun 6 11:59:39 xxxx kernel: [3015924.308771] device-mapper: cache: 252:3: promotion failed; couldn't update on disk metadata</div><div>Jun 6 11:59:39 xxxx kernel: [3015924.308846] device-mapper: cache: 252:3: metadata operation 'dm_cache_insert_mapping' failed: error = -22</div><div>Jun 6 11:59:59 xxxx kernel: [3015944.161468] device-mapper: cache: 252:3: promotion failed; couldn't update on disk metadata</div><div>Jun 6 11:59:59 xxxx kernel: [3015944.161541] device-mapper: cache: 252:3: metadata operation 'dm_cache_insert_mapping' failed: error = -22</div><div>Jun 6 12:00:01 xxxx kernel: [3015946.457345] device-mapper: cache: 252:3: promotion failed; couldn't update on disk metadata</div><div>Jun 6 12:00:01 xxxx kernel: [3015946.457416] device-mapper: cache: 252:3: metadata operation 'dm_cache_insert_mapping' failed: error = -22</div><div>Jun 6 12:00:42 xxxx kernel: [3015987.290496] device-mapper: cache: 252:3: promotion failed; couldn't update on disk metadata</div><div>Jun 6 12:00:42 xxxx kernel: [3015987.290569] device-mapper: cache: 252:3: metadata operation 'dm_cache_insert_mapping' failed: error = -22</div></div><div><br></div></div></div>